New Test Release

[cavemanbob]

This is a test version with some simple new client pull code, in contrast to the old DOS server spam model :p It may or may not work, but that's why it's a test. This will hopefully fix some of the random stop working bugs...


http://alteria.sf.net/myseq-test.zip
http://alteria.sf.net/myseqserverc-test.zip
http://alteria.sf.net/seq-maps.zip

Server Changes:
* Update command line parameter now does nothing. This is handled on the client.

Client Changes:
* New Update option in the Options Dialog. This is the delay to wait between receiving a stop packet from the server and sending a request for the start. If this is set to 0 it will go as fast as it can, limited either by the client processor, the server processor or the network, whichever is slowest. It can update really fast at 0 though.

And of course if there are problems, please report them

[MrDoh]

Caveman,
I'm sorry to double post this, but now that it's in an old thread (guess we were posting at same time) I didn't know if you would see it, and kinda want to know if my logic is anywhere close.

Just a thought about this...

----
Lemme know if it won't work, but there should be a "formula" that can be worked out that will.

You set how fast the server reads the memory updates. I see no usefull purpose on polling the server any more often than the server itself updates. Would it possible, (or even beneficial at all) for in one of the very initial packet exchanges, have the server send its update speed to the client, and let the client start with that poll the speed (this is where that formula comes in... maybe have to add a few ms for the server to get everyting updated before it sends it). Then, if the users wishes, they can modify the update time more to their liking. If 250ms is to fast to send the packets out (1/4 second shouldn't be too fast I wouldn't think) then what's the point of having the server read that fast. And if the client is set to update faster than the server, you are causing unnecisarry network traffic because nothing will have changed because there hasn't been an update. (read duplicate packets)

Just some thoughts,
Mr. Doh

[cavemanbob]

Interesting idea, but in the current scheme the server won't poll more often than the client does. In fact the memory reads are directly linked to the client's requests until I work out a better solution. I'll think on this though, it could be a way to avoid at least some client -> server communications.

[cavemanbob]

Out of curiosity has anyone that was having trouble with the serevr locking on sends had this version work better?

[MrDoh]

Well, here is what I have found. So far, it has fixed my intermittant lockup problems in pre-LoY zone. (At least it hasn't locked up yet, but I have not been in game very long.) I do think that problem is fixed, based on what was thought to be happening.

Now, back to the dreaded LoY lands hehe.

Not much has changed. I zoned from steamfront, and it didn't even finish clearing the spawn list! It also didn't change the map.
If I start it up from inside gunthak, it does still load the map, still shows the location I am standing when I started the client, and at present there are even 3 skittles other than myself on the map. There is nothing at all in the spawn list.

When I "stop" and then "go" the client, nothing changes on the client side. (The server side still does what it should.) It doesnt update my position, the same 3 other skittles are all still there and in same spot. Well no sooner than I say that, this time when I pressed stop and then go, it did redraw part of the map (over part already still there), it did move me to mey new loc, and there are a few more skittles. Still nothing in the list, and not updating loc, etc.

I am going to check and see if you left the debug features in the current build. I'd like to capture the spawn list of gunthak from the server and see what is there. Is it possible that *something* in the spawn list of gunthak is messing with something in the client? I wouldn't think so, but there is no telling what SoE may be trying in LoY.

Anything else you can think of that I can do? Is there a way I can dump 1 set of packets to a txt file for reference, then maybe dump a non LoY zone and do some down and dirty comparisons?


Mr. Doh

*edited for more detail and more testing

[cavemanbob]

I think I'm just going to download LoY over the next day or 2, it'll only take 9 hours so I'll just leave it at night. There's a bit of code in the server where it sends the spawns to the client to dump the first mob to a fire called mob0 commented out. It's fairly easy to adapt to dump all of them, just stick it before the send and change increment the number in the filename. Unfortuanately the debug code was removed, but it was just some printfs around the read/send blocks anyway.

[MrDoh]

CavemanBob, please check your PMs if you would.

thanks

[MrDoh]

Most of my client lockups continue... My lockups (hard freeze, end task) seem to mostly (but not absolutely always) occur when I'm actively working with the interface, not when I'm running and not actively using it and seeing everything update.

I can still say that the pull solution you have implemented seems to work very nicely.

It's still interesting to me that the client doesn't lockup with the LoY problem. You can still "stop" and "go" and use the menus, etc. It just doesn't give a spawn list, or continue to update the client after that initial spawn list never happens. But yet I have seen the spawn list with a hacked up debug version.

For what it's worth,
Mr. Doh

[cavemanbob]

I've got to admit, I have no idea what could be causing lockups by using the UI. Is there any particualr one that gives the most trouble?

[Alfred]

My feedback on new client server handshaking:

Again, I'm using a 300mhz PII for the client.

1) If I leave the client alone it will work fine for a long period of time before locking up.

2) If I start changing things, clicking or doing many other menu related tasks on the interface, it locks up.
A) I attribute this to buffer overrun. The client sent out an "its ok to send" packet when in fact I started clicking and took cpu cycles away.... I have no idea if this is in fact the truth.

3) The very first versions that did not have the speed lines updated the screen much more quickly. In fact, before you made this test version with the handshaking, I couldn't get it to work no matter what delay I used. Now I realize that all the extra features are simply to much for my little old box to handle hehe. It looks like I've got a delay of about 2500ms when in fact I've tried 0 through 1000ms :).

soo... the hand shaking route is the way to go.. .Now I've either :

1) Got to upgrade my hardware (on my 4th machine /sigh)
2) Go through the code and see if there is a way to introduce a toggle for extreme bare minimium functionality for stupidly slow PCs. ;) I'll have to think on that one....

Anyhow, great work so far. I don't know if we are out of the woods yet with the lockups but this is a step in the right direction.

cheers :)

[MrDoh]

Well, I think Alfred has some very good points and observations.
The lockups do mostly seemto happen for me when actually clicking buttons, map skittles. The zoom and X, Y buttons do it to me a lot, but then again, I use them a lot. Sometimes it happens moments after you start the program, sometimes it may be hours, depending on what you are doing with it. But just now as I type, I tabbed to a different window (it wasn't active to begin with) and it froze. It was never even the active window, much less clicked on.

As for my hardware and O/S, the client is on 440mhz PIII with 128 mb ram running NT4SP6a with .NET framework 1.1.

Runs very smoth on this machine, even at the default 250ms (fast as I think anyone really needs hehe). Which as said in other posts, it looks amazing, and real time!

What type of network packets are you sending? What protocol are they based on? How does the client-side pull actually work? What kind of packet flow control is there? What size packets are being passed, and are they being passed one at a time, and then the next update time, requested and passed again, or is it broken up by the different offsets, or are all packets a certain size?? What is a *maximum* packet size??
(I'm a network engineer on the hardware side by trade, with other interests in PCs, so everything is seen through slightly tinted glasses.)

Please let me know please about the packets so I can get them off the wire and look at them. My normal sniffer doesn't seem to understand them, and isn't letting me snag them... hmmm...

This is a wonderfull program and getting better by the day, we just still have a few things to work out.

[Dark]

I used the test client and it started fine but everything stopped moving after about a min. If i closed it and started again same thing happened.

My eq machine is a 2.66 adn client machine 1.7gig both with 512 ram.

I rolled back one version.

One thing that would be helpful is the version number in the top blue bar.

[cavemanbob]

Ahh, this may very well be a performance issue then. I'm DLing a .net profiler now so I can see what is sucking lots of cycles.

The communications happens as follows:
The protocol is TCP, UDP has less overhead, but is more of a pain to use. All sockets are blocking and because it's TCP delivery is guaranteed.

All packets are 96 bytes long regardless of function and are of the SPAWNINFO_SEND structure, this is just for simplicity. I'm not sure of the timeouts now that I think about it, I'll have to look at those.

Here's how the data stream goes:

Client: Packet request -> Server
Server: Start Packet -> Client
Client: Packet request -> Server
Server: Target Packet -> Client
Client: Packet request -> Server
Server: Player Packet -> Client
Client: Packet request -> Server
Server: Zone Packet -> Client
Client: Packet request -> Server
Server: Start Packet -> Client
Client: Packet request -> Server
Server: Spawn 1 Packet -> Client
Client: Packet request -> Server
Server: Spawn 2 Packet -> Client
.
.
.
Client: Packet request -> Server
Server: Last Spawn Packet -> Client
Client: Packet request -> Server
Server: End Packet Packet -> Client
Client: Sleep for n ms for a delay, then repeat

If there's packets bigger or smaller than 96 bytes something in the stack has broken them up or combined them, it's not really important. As another thing I just thought of, in some cases the sent packets might be getting buffered for a while on the server side... Combining all the little bits into a big packet is starting too look like a better and better idea.

I've put the version number in for the next one, I meant to get it in the last one, but forgot.

[MrDoh]

I don't know, you might could call it a performance issue, but I think it may be a timing issue. I've been watching, and my CPU only maxes out to 100% for about 4 seconds when you first hit the go button. Even when it gets the update, it only peaks at 30%. When using, say the zoom buttone, it uses no more than 45% or so, and only for the second or so it takes to redraw the map. It just froze on me, and it was at 48% cpu usage. When you really come down to it, I wouldn't expect the client to require a large cpu at all. We aren't drawing complex textures, or doing immersive 3D etc. No severely complex calculations, right? I would expect something like this to run like a top on a 450mhz machine. My client machine is only a P3 550 running xp /w 384mb ram, and it runs just fine with myseq server, outlook, several browser windows, etc, and the only time it if slows down EQ running with high graphics is in the bazaar and pok.

Oh, and the network structure looks great. Using tcp is definately the way to go here imho. One question, when the client recieves the packet, does it immediately dissasemble and parse it, etc, or how? Is it still trying to take care of packets that's already arrived while striving to complete processing on what it's got. Yes, that is set by the prefference as to how long to update, but if it isn't finished withe the last set, or it falls "X" number of packet sets behind, should we through in a warning, or a pause to complete processing? To paraphrase a line in "Four Rooms" , I'm not a bunny, and you're not a rabbit, so lets not jump ahead. hehe

I really don't know what the issue is at this point... but at the same time, I have no doubt we'll track it down.

Mr. Doh

[cavemanbob]

The client immediately pulls the packet apart before asking for another one, so the server shouldn't send another one till the client's done with the last one, so it shouldn't get ahead, but I'm not absolutely sure this isn't happening. I'm going to throw together a different version of the client & server to take one big packet anyway jsut to see what happens anyway though.

[MrDoh]

I just did some more basic tests. If you would compile me a version based on this test code version, and just for testing purposes, put a pause in the updating while the mouse is being presses. (please, please, please... and I'll test 2 night)
(But get your sleep etc... I'd rather wait than you get burned out!)
hehe
My thought is, if you do this, I will not have many more lockup problems while zooming.

What is actually making it freeze though? Just because the cpu hits 100% for 2 seconds is no reason to just up and quit, is it?


Hey wait.... maybe the client, or the server needs to decide to throw a way a few packets if it gets over a threshold... Still just poking in the dark here...

There again, are we looking in the right direction? is it maybe the screen trying to redraw or rearange in the middle of something else going on that freezes it. But that something else only happens when connected, because when offline and I load a map I can zoom, and move the map all over, use all the menus, and columns etc, so it is something that only happens while connected.

I will tell you that offline, with a map loaded, *holding* down the zoom button all the way to 10000, and immediately back down to 100, Iand cpu at 100% can do this repetedly without problem. With the same map loaded and connected, the most I have ever gotten to was 250 zoom, and the cpu never even got to 100%.
I even go, then stop, then zoom to 10000 fine.

For what it's worth,
Mr. Doh
As a side note, tell me what you code the client in? I know it's C#, but do you use an interface, do they call it an IDE? (hehe told ya I'm no programmer!) If I get a little more information and get quazi-structured look at the client code, I might could be doing tonights testing and recompiling without wasting your valuable time. Really, thanks again for all this hard work!

[cavemanbob]

I use visual studio .net, which the best IDE that i've had so far.

I've got the code for using big packets done and am just giving it a quick test before posting it. It seems more responsive at first glance anyway, which isn't all that suprising.

The freezing could still be due to a threading issue, I really should double cehck and make sure that all shared data is being locked when it should again.

I'll look at the update pause thing, it's just a matter of pausing and restarting the net thread, but I'm not sure how to capture this for everything in the form. As another note, by far the most time is spent in the drawing function, ~26%, while the netthread comes in second at ~6%.

[cavemanbob]

The big packet thing works perfectly, and I tracked down what I think might have been causing most of the problems in the 1.8 release. It seems that the X Y Z coordinates I added to the listbox take a REALLY long time to update, after removeing the updates on those it runs way better. A new test version will be ready tommorow around noon cst.

[Sixes]

I can't help feeling your going down the wrong path with this client pulls paradigm.

What were you trying to fix? What was wrong with the server push method?

[lane]

Just a note on the Test app. Been working fine for about 2 hours now concurrent with MQ as well.

Client and server both running on notebooks over a wireless lan, wiht maybe 512 meg in one p'uter and 384 in the other. Win2K pro on both.

-Lane

[Proxeneta]

It seems that the freezing problem clearly only occurs on systems running pre winXP. I am running WinXP home for the client, and It seems to run just fine. But when I try to use it on Win98 I get immediate freezing as soon as I start to click on skittles, ect. I suggest using Winxp for the client with v1.8 and a winxp or below server with 1.5

[MrDoh]

Proxeneta,

Using all XP would certainly be wonderful, but isn't going to be practical for everyone. If I had a second machine that I could run XP on, believe me, I would be. If I had the mney to go build another PC, it would be XP, just like I run on the server/EQ PC.

My lockups were happening just as frequently with the pre 1.8 codebase.

This is a .NET application, so it should not *require* XP, but rather require .NET, which is available for many win-based platforms.

And, I also must respectfully disagree with your client-side pull observations. If the server just "spams" all the packets with no control from the client, I think you would be asking for trouble. At least with the client-side pull, there is some sort of flow control, which I believe did not exist in the "spam" code.

Just my .02

Mr. Doh

[Alfred]

Originally posted by Sixes
I can't help feeling your going down the wrong path with this client pulls paradigm.

What were you trying to fix? What was wrong with the server push method?

For slow PCs, the server push method simply wasn't working with the client design as it stood. Would lock up tight. Now it works... very slowly but it works.

Now that seq is working, and linux simply isn't a big of a resource hog, I won't have to struggle with a .net app on a PII 300mhz.

But keep up the good work CMB. :)

/cheers

[cavemanbob]

-- I can't help feeling your going down the wrong path with this client pulls paradigm.

What were you trying to fix? What was wrong with the server push method? ---

The problem with this method is that on slower PC's they appear to be able to ovverun their internal TCP buffers at which point something bad happens and the server locks on a send. In the end I like this setup better anyway, if I set the delay to 0 it updates really smoothly, it uses a LOT of network bandwith, but I don't really care.

Using XP is completely unnecessary, I have never started this application on XP. I run Win2k sp1, Win2k sp2 flawlessly and occasionally win98, but it doesn't work as well.