MySEQ Server 2.0 Sneak Preview

[MQSEQ2]

SNEAK PREVIEW

Here is what I have so far. It dosen't do alot but you will start to see what it looks like.

http://www.dvolve.net/EQ/MySEQ/MySEQServer2.0SneakPreview.zip

You can connect to via telnet by going to a DOS prompt and typing:

telnet {Your Server IP Address} 6969 [Enter]

That should get you a blank screen and if you hit [Enter] it will give you a Connected! message in the telnet session.

If you watch the Server Window it will show you the status of what it's doing.

Close the DOS prompt and it will close the connection to the server and the server will reset waiting for another connection.

Please check it out and give me some feed back on whether it works on Win98/WinNT etc.

I will continue to add more functionallity to it today.

[AriB]

Sweet Looking good

[MQSEQ2]

I will be releasing a new sneak preview of the Server 2.0 tomorrow once I get some more items `done.

Currently it will scan for EQprocesses and add up to 5 EQProcesses in a menu so you can select which process to monitor (it will only show you the ID not the players name). maybe once I get it done I will have it hook each process to get the players name and display the name vs. the Process ID (sounds pretty cool, glad I have a big check book :)).

This is a total rewrite of the server and once it's done it will be alot faster and will be customizeable of what data to transfer. You will get to select what is set to the client, ie. SpawnInfo, GroundItems, PlayerInfo, TargetInfo, etc. This will allow you to toggle what you want to see in the client. Basically if you are running slow then you can turn off Player info, grounditems, etc, this would reduce the amount of traffic to send which would mean the faster screen refresh, so break put the P75 and knock the dust off. Dang P75 wont load any OS hardly hehe.

[MQSEQ2]

Here is the latest Sneak Preview.

http://www.dvolve.net/EQ/MySEQ/MySEQServer2.0SneakPreview.zip


You can run this version and it will add a list of EQProcesses to the menu for you to select. Once you select the EQProcess you will see the Zone Name in the Titlebar.

You can telenet into the server via Telnet on the IP Address and Port number (Port is in Ini file). Once you are Connected you will see the Server show the IP Address and Port that the client is connected on. You can do the following tests:

Key Server Response
Enter Connected!
0 Spawn Info
1 Target Info
4 Zone Info
5 GroundItem Info
253 Player Info

To do 253 you will need to use a line mode Telnet program like PuTTY (free download). DOS Telnet is a character mode Telnet session. Charter mode is it send each character vs. the entire line once the Enter key is pressed.

Please check to see if you see those outputs. The program doesn't do anything else at this time but as you can see I'm making pretty good progress on this rewrite.

I need folks to test this on Win98/WinME/WinNT/Win2k/WinXP.


DEVELOPERS:
I need some help with the section in frmMain.cs labeled ToDo:.

Basically I wrote what is needed in the comments.

[Mixy!]

Pardon my noobness, but can you explain the telnet thing to me? What will that do for the new server? Is it a different way of connecting to it in leiu of the client?

Or is telnet just being used to test it at the moment?

-Mixy!

[MQSEQ2]

Telnet is just for testing purposes You could telnet in the existing Server.

A Server/Client setup a Port to communicate via TCP/IP (hence the IP Address) and port. The client will ask the Server for information across this Port. The server will gather the rrequested information and send it back to the Server via the port.

The Client you are using is just a fancy Telnet Client that sends request for data and then presents it to you in a way that makes sense. EQ is based on this type of theory as well.

[CybMax]

You can run this version and it will add a list of EQProcesses to the menu for you to select. Once you select the EQProcess you will see the Zone Name in the Titlebar.

Hmm.. What does this mean? I start it both before and after i started EQ, but i did not get any options or "menu's" to choose anything tho.. hehe

Well.. Looks kewl nuff.. Keep up the work :)

[MQSEQ2]

The menu I'm refferring to is the menu that pops up when you click on in the SysTray. Generally all SyaTray apps work off that menu system, most SysTray apps don't have a window to display hence SysTray Icon.

In the Final product I will add additional Menu options so you can see them when the Window is up. I will also be adding a Links options and allowing you to have 5 web site links that you can modify and point to whatever you want.

This is the time to make some request for the Server as well as the Client. I say the Client here because some Client requests may require Server changes to push the data being requested. Think along the lines you see it in EQ and you would like to see it in this project.

[reaver]

Would like to have a data dump of your character to a .txt file.

All equipment worn, in inventory and in bank along with it's ID#, name, and Lore descriptive.

This is all info that's sent, as I know I've done this before in SEQ.

Great to be able to print your character items out for reference, etc.

-Reaver

[MQSEQ2]

Inventory stuff maybe more than what MySEQ is used for at this time. MacroQuest does this type of stuff already.

I plan on showing the stuff that you see in SEQ. As you see in SEQ you don't really see the name of the worn item just the class.

I will try to get all the data I can tho.

[Mixy!]

As i posted in the other thread, i would like to see ZEM's displayed somewhere if possible.

-Mixy!

[z26o]

A client feature I would like to see, which would probably require a server mod is the spell casting info window. Where is list the spells you have casted and the time remaining on them.

Great job on the project! Thanks :)

~z~

[ap50]

Originally posted by reaver
Would like to have a data dump of your character to a .txt file.

All equipment worn, in inventory and in bank along with it's ID#, name, and Lore descriptive.

This is all info that's sent, as I know I've done this before in SEQ.

Great to be able to print your character items out for reference, etc.

-Reaver

You can do all that excepting the lore description with the Magelo client.

[MQSEQ2]

ZEM is on the list.

The Spells buffs I will look at, I know MQ does this show I show be able to figure it out. I will try to make a window showing buff info.

[MQSEQ2]

Major breakthru in the rewritting of the Server to C# today.

I have been able to get more data from the client today, this means the new server should be released soon. I will release another Preview today hopefully that will be able to return all the existing SpawnInformation as well as some new stuff.

I need folks to really test this product ASAP on all the OS Platforms becuase once it appears to be working fine the Client will be modified to work with the new Server. So I'm trying to work out any OS issues before we make Client changes.

Once you have tested it on your OS please post it here working or not.

We need the following OS's tested:

Win98/WinME/WinNT/Win2k/WinXP

This is everybody's chance to contribute to the project outside of developing the project. The developers are working for free to make this product a better project. All we ask now is help from folks on testing and helping us debug the issues.

Thanks for your support!

[Fidd]

Just curious how this are moving along... I'm really looking forward to this server :)

/Fidd

[MQSEQ2]

I'm going to post the latest code today. I will need to do time checking on sending SpawnInfo. Currently I'm sending each Spawn as a separate packet (Approx 500 bytes) to the Client. I may change this to build one large data stream then send it over to reduce the number of packets. Or may add 3 Spawns per packet.

The One big Data Stream would be better if I can find a way to compress the data down to a smaller size. You can do the math of number of Spawns times Packet Size to determine Data Size / by 1500 (approx Packet Max Size). Ex. 400 Spawns * 500 bytes = 200k bytes / 1500 = 133 packets. Now if I could take that 200k and compress it to 100k then it would take hale the pakets to transfer the same amout of data.

[jag111]

Since you're trying to turn the server into pure C# code and assuming the performance is good enough to handle it, you might consider other ways of getting the data from the server to the client.

The way in particular that I was thinking about would be to use .NET Remoting. The server would basically be hosting a remoting object that is collecting the data from memory and doing some *small* analyzation of it (i.e. check what has changed). When the client connects to the server, it would basically just instantiate an instance of that remoted object and call methods however it wanted. There are a variety of benefits to this:

- Zero networking code. We can completely get rid of it. Remoting takes care of it all for us. Just a few lines to start up the remoting stuff on each side and we're golden.

- The "conversation" between the server and client enables us to be more efficient in processing the game data. Let me explain in further detail:

Conversation right now:
-------------------------------
Client says, "Give me everything"
Server says, "Here you go"
Client not only has to process all the data and see what's changed, it then has to make it visually appealing on the client.

Ideal conversation with remoting:
-------------------------------------------
Client says, "Give me everything that has changed"
Server says, "Here is this structure with the 80 spawns that have changed" (out of the 500 in the zone)
Client now only has to process 80 spawns and already knows that they will be being replaced in the list.
Server says, "Oh, btw, you've changed zones. You're now in Everfrost" (via an event that the client is subscribed to)
Client thinks, "Thanks, give me all the spawn info"

There are so many possible things we can do instead of just the old tcp spew of the current spawn state.

We could theoretically even have server sided filtering that could lower the cpu utilization even more. Like when the client starts up, it could tell the server that when determining if a spawn has "changed" ignore the Class, Race, and Run Speed (which I believe we currently have just hard commented out right now). But this way people could choose what fields they actually cared about.

Forgive me if I've been a little long winded about this. I was considering just keeping it to myself until the first revision of the 2.0 server was out and sort of doing it myself. But it's gonna be a pretty large code change due to the sort of fundamental changes in the way things will happen. I'm also looking for any possible downsides to the idea.

So hit me.

P.S. If the remoting idea seems scary because you're not familiar with it. Trust me, the code that remoting adds is quite minimal. And once it's in place, you can pretty much forget about it. From then on, it's just writing methods and events as if the client and server were just one big application.

[MQSEQ2]

I was looking at doing it via Remoting so that would be something to look at. I will dwell some more in that area.

I guess I need to stop playing EQ today long enough to zip up the latest SP 2.0. I'm successfully in gather all the data from the 1.9b server as well as all the new stuff that I've put in the Ini file.

I would love to make it so it only sends minor updates to Spawns that need it. That would drastactly reduce the data push from the server making the client very responsive.

So basically, I ill need to create the Mob structure (borrow from the Client) then do the compares on the server. Thought process is to set a flag in the structure whether it's a new Spawn or an update (this would prevnet the need to check in the client).

I will work on this stuff this coming up week and I'll create a small test client to verify the data stream. This would also help use in the future with troubleshooting.

[MQSEQ2]

Get the latest SneakPreview at:

http://www.dvolve.net/EQ/MySEQ/MySEQServer2.0SneakPreview.zip


This version is connecting and pulling the information from EQ and sending it to the telnet client. There is no optimazation in there so the Spawns may be a little slow (I haven't compared times between 1.9b vs. SP 2.0). I have found some In-Memory Compression routines that I will be adding in this week.

I'm also looking into the .Net Remoting (the old DCOM) to see what I can do there. I will add the feature of only sending the modified information. This will help decrease the number of packets being sent.

[MQSEQ2]

I've started the In-Memory Compression testing and here's the results so far:

SpawnInfo Structure Size: 437 bytes/Spawn
After Compression Size: ~160 bytes
Compression: ~65% smaller

This should help out alot when we have 400 Spawns in a zone.

Examples:
Without Compression:
400 Spawns * 437 bytes = ~174 Kbytes / 1500 bytes = ~117 packets

With Compression:
400 Spawns * 437 bytes = ~174 Kbytes * 35% = ~61 Kbytes / 1500 bytes = ~41 packets

Now once we add the Server side logic to only send Spawns needing updated then those numbers will be reduce even more.

[reaver]

Main thing I'm running into on the client computer is the CPU usage. Even with the enhancements that have been done in these newer versions, it's still quite laggy (mouse jumps all over or barely responds at all).

Not sure if the compression will increase CPU time (to decode the compresses packets) or not.

I have enough bandwidth on the network - 100mbit, just not a fast processor/memory for the client.

Just wasn't sure the final goal of the compression changes. :)

Thanks

-reaver

[MQSEQ2]

The In-Memory compression/decompression should not be noticable due to the data is already in memory. It's not like running WinZip and decompressinga .Zip, that's because in a .Zip file you have to read the file from Disk into memory then once it's in memory then decompress it. The time delay is the harddrive access not the memory part.

As I stated in another post, if I send less data (because of compression) then I don't need as many resources (buffers) to store all the incoming packets. When you have alot of buffers it can cause higher CPU ultilization becuase if your computer doesn't have enough memory it will swap it to the swap file and read it back into memory when it needs it.

[MQSEQ2]

Here is some of the results from reading the Spawns:

With 4 HiResTimer check:

Read Memory Duration: 3.5376ms
Original: 348800: Compressed: 50632 Spawn Count: 800
Compression Duration: 83.1038ms
Send Packet Duration: 38.3071ms
Overall Process Duration: 125.2112ms
-------------------------------------------------------------------------
Read Memory Duration: 3.8267ms
Original: 348800: Compressed: 50554 Spawn Count: 800
Compression Duration: 86.1467ms
Send Packet Duration: 31.5845ms
Overall Process Duration: 121.7442ms
-------------------------------------------------------------------------
Read Memory Duration: 3.3638ms
Original: 348800: Compressed: 50324 Spawn Count: 800
Compression Duration: 87.0255ms
Send Packet Duration: 36.9932ms
Overall Process Duration: 127.5754ms
-------------------------------------------------------------------------
Read Memory Duration: 3.7742ms
Original: 348800: Compressed: 50132 Spawn Count: 800
Compression Duration: 80.3697ms
Send Packet Duration: 39.4749ms
Overall Process Duration: 123.8068ms


As you can see the Avg Read/Compress/Send is ~125ms with the 4 HiResTimer check.


With only 1 HiResTimer check

Original: 348800: Compressed: 49990 Spawn Count: 800
Overall Process Duration: 74.7514ms
-------------------------------------------------------------------------
Original: 348800: Compressed: 49915 Spawn Count: 800
Overall Process Duration: 71.905ms
-------------------------------------------------------------------------
Original: 348800: Compressed: 49913 Spawn Count: 800
Overall Process Duration: 78.3619ms
-------------------------------------------------------------------------
Original: 348800: Compressed: 49902 Spawn Count: 800
Overall Process Duration: 75.5518ms
-------------------------------------------------------------------------
Original: 348800: Compressed: 49897 Spawn Count: 800
Overall Process Duration: 92.5489ms
-------------------------------------------------------------------------
Original: 348800: Compressed: 49866 Spawn Count: 800
Overall Process Duration: 77.1447ms

As you can see the Avg Read/Compress/Send is ~77ms with the 1 HiResTimer check.

The HiResTimers will be removed prior to release, I used them for Proof of Concept to make sure I can get the data where I need it in a certain timeframe.


There was ~610 Spawns in zone and I had preallocated enough memory to handle 800 Spawns. By doing this it greatly increases the Read Memory times.

I'm trying to locate the actual SpawnCount prior to intializing the buffer so I don't have it hardcoded (obvious reasons), but if I have too, I may set it to 1000 (~436 Kbytes of RAM).

Another option is, if I can't fina a Count I cycle thru all the Spwans once to get the count then run through it agian to read it for transferring. The only issue I see is if I run thru once I get 466 Spawns and on the next run thru I get 467 Spawns populated. Based on Read Memory Duration: 3.3638ms the chace is very small but could happen.

[jag111]

Since you've already got the code in there, I'm curious how the Overall Process Duration changes if you take out the in memory Compression.

I only ask because it seems that of the 3 stages, the compression takes the longest to do....at roughly twice the length of the send. So basically, is the extra cost of having to send more data over the wire worth it in cpu time by taking out the compression step?

Because, as a couple people have said, *most* of the users of MySEQ are going to be running this over a local LAN where bandwidth really isn't a problem.

[MQSEQ2]

Let me write a test client so that I can get the exact time from the Clients PoV. The times listed above was in one routine that Read/Compressed/Send.

When I do it without Compression I'm getting the following results:

Overall Duration: 16ms
Overall Duration: 23ms
Overall Duration: 10ms
Overall Duration: 47ms
Overall Duration: 5ms
Overall Duration: 25ms

The times are suspect tho and that's why I want to build a Client that can time it from end to end and let the Server time it internally as well.

If I don't need compression then that's fine with me.

[jag111]

It wouldn't surprise me if those times turn out to be pretty close to accurate.

Perhaps, though, for the few people who are actually trying to run this over the internet rather than a local LAN and do care about the bandwidth needed for the data, we can just have it be configurable to either have compression or not.

[MQSEQ2]

I can add the compression as an option.

Whats made the big difference is the pre allocate memory.

The reason I don't trust thos numbers is because the test was done with PuTTY and also DOS Telnet and the DOS one scrolled for another 8 seconds or so (which was shown in the 4ms times). The 348800 was stored in a buffer then sent via when the client could handle the stream.


It the times with Compression on I tested with both progs and the progs stop scrolling the info very quick.

This is why I'm going to create the MySEQTester for timing test as well as troubleshooter for the products in the future.

[jag111]

Sounds like a good plan.

[MQSEQ2]

These test came from the MySEQTester program being written for the project.

Here are some times test without compression:
(Client Request/Server Processes/Client Recived)

Duration: 134.745ms
Duration: 145.364ms
Duration: 173.210ms
Duration: 207.984ms
Duration: 142.757ms
Duration: 143.089ms
Duration: 153.332ms
Duration: 166.394ms

Now there is a little bit of overhead becuase of the HiResTimer functions, so the actual times are better. I have seen the numbers drop down to 40ms.

Looking at the Current Client Source Code, the delay is between the requests, so it doen't matter how long it takes to get the data from the server because the logic checks if their is a data request.

The tests above was in the Bazaar with about 580 spawns. So we should be able to progress forward without compression (I will continue to do more testing with compression in the future).

[MQSEQ2]

A new SP 2.0 has been release at http://www.dvolve.net/EQ/MySEQ/.

There is also a test program called MySEQTester that is a stripped down program showing you the new features that will be coming.

Verify the Ini file has the following:

SpellsAddr=7453592

[stranger]

Why aren't we looking at what it would take to take the SEQ Linux code and port it to windows?

Seems like this would be a better choice in the long term and could be used to bring the two projects together.

SEQ could be seperated into a sniffer and a UI. As long as the sniffer was writen with no ui (or just standard io) it should work fine with both versions. Then the only thing that is seperate is the GUI to display everything.........

Just a thought.

[MQSEQ2]

I have some code that will allow us to make the NIC go into promiscusious mode (listen to all packets).

They are also using code that is unix based only and all that would have to be redone. Then you have the whole Linux vs. Microsoft battle.

My goal is in the future is to make the Client be able to go into Server mode then later on take the Server totally out (way down the road).

[MacQ]

Get rid of the server...that would be so cool! The client and server in one program on a machine seperate from the system running EQ...I love it. So glad you are thinking about heading down that path.

[MQSEQ2]

And when I really get good I will rewrite the entire Project with five lines of code :D. Yeah Right, but the Client/Server will be coming as soon as I'm done with the 2.0 server. I'm trying to make the code modular so I can Plug and Play (this isn't Microsoft's Plug and Pray either).

[magictiger]

As strange as it sounds, I REALLY like having a seperate client and server. This allows me to use a switched network and still be able to run the client on one machine with the server on another. If you could find a way around that, then it would be great to put the client and server into one program. I really don't see how you could do that though.

Of course, I'm more of a beat-it-with-a-wrench technician than a coder. :)

[MQSEQ2]

Until the Project can do promiscusious mode (listen to all packets) the Client will need a Server if the Client is not ran on same machine as EQ, that's why we have the Server. I have read alot of folks that currently run the Client /Server and EQ all on the same machine so my plan is to make the Client be able to access the EQ memory directly without the need of a Server. This will be an option for people to choose for themselves.

[cavemanbob]

I like having a separate client and server too, that's the biggest reason I did this in the first place. I too think that not having ShowEQ work just because I wanted to use my 100Mb switch was a bitch. Having an option to run a windows client like the sniffing linux one would be neat, but that'd be a bitch to implement too and it'd be largely pointless, might as well just write an alternate server to do it. I thought about doing so at one point in the past and figured it was stupid anyway since SOE doesn't seem to care about people reading memory anyway.

[jag111]

/agree cavemanbob

Not to mention, you'll always theoretically be able to glean more information from memory than a packet stream anyway because there are things that the EQ client just takes care of itself and never needs to talk with the server about.

[MQSEQ2]

What I was planning on is to give the user the option to run as a Client and Server (just like current) and give them the option to have the Client read the memory (not promiscuious mode).

It would be a very long time be I even attempted promiscuious mode even tho I have the code that could do it. My goal is to get all the features into the Project be we do away with anything.

Hint SPAWN TIMERS are working great and will be released today once I get the flashing (Yellow, Red) coded but everything else is working great.

[Tbird038]

Is the new server ready for Prime Time?

Tanks!!!

[MQSEQ2]

It's close to prime time, now I need to rework the Client to use it. I will be releaseing a Client today and then I can start on the new Client setup.

I'm trying to get more features added into them and then all development will need to be put on hold for a week or so , so I can get it all working.

[CybMax]

Definately working good as it is with a "server" client running on the EQ machine, and the "Client" running on a separate machine.. (At least i would not have it any other way)

The ShowEQ way of doing it by listening to all packets, and decoding it in only the client adds both positivity and negativity..

Positive : You dont have to run anything on the EQ machine, and there would be NO way for SoE to discover you using any spesific program to do this

Negative : As most people now have switches even at home (due to hubs are obsolete now), the packet sniffing way is much harder. I got this to work by setting up the linunx box with 2 nics, and using it as a internet router.. ie. EQ -> Linux -> Internet. No big problem really, and it worked very good, but i like the MySEQ way of doing it a lot more (bah.. my "linux fanboy" days are over for good.. Now even RedHat going comersial as well..)

So.. Keep it the way it is with server<->client setup..

[MQSEQ2]

The only thing I kept from Linux was my lil Devil friend hehe.

If I do promsicious mode it would really only be for the challenge and learning of something new for me to do. I love to reverse engineer things so it would be fun to learn.

If it did go to PMode we would still support the other ways as well, it would just be more options for folks, but I don't see me attempting it for a couple of months, reaaly depends on how bored I'm at work.

[WyvernX]

If some of you want a client/server and others just want a single client, this is the easiest solution.

Have a single client application that acts as BOTH a client and a server.

Ie, you run the application on one machine, and hit a server checkbox.

Run on it on another machine, and hit the client checkbox.

Run it on a third machine, and it defaults to client/server.



Just need to do a quick port of your cpp to c# and you're just about done.

[MQSEQ2]

The way I'm going to do it is basically the same way we are doing it now with a Client and a Server. But then I'm going to take the code from the Server and add only the parts I need to read Direct from memory and a Server won't even be needed if it's on the same computer.

[Jono]

Is there any possibility to move the whole program from C# into C++?

I hate having to install 20MB of support files just to run a 300k program.

If that is not possible could a version be compiled that doesn't need the .NET support stuff?

[MQSEQ2]

You could compile in C++ if you didn't mind rewritting code that uses the .NET features.

C# 2003 is intwined with .NET 1.1, you can thank Microsoft for that. It's Microsoft way of not being accussed of being a Monoploy. Just like Internet Explorer is part of the OS you can remove it. Well C# requires .NET and soon .NET will be integrated into the OS (Win2k3) and you have no choice. Long live Bill!!!!

If you asked me I would make it a VB 3.0 project and just make it Thunk the 32 Bit API calls. Then we would have a fast program hehe.

[jag111]

In a word, no.

As for having to download 20 meg of an "application framework", deal with it. .NET is the future of Windows based programming. I know I already run 3 other apps on a regular basis that use it, one that I wrote myself. The next version of windows will most likely even have it embedded. Get mad at Microsoft if you like, but as a primarily windows based programmer, I frickin love it.