HOWTO: OpCodes, Structs, package de-serialization, etc (yes, again)

[tanner]

Spent the last couple days searching and reading the forums, looking to collect in 1 spot the info necessary to help fix seq on patch day.

This question has been asked many times over several years, but all the threads just seem to fizzle out. Many of the past threads say:

there can be no "real" HOWTO guide to start you out on this road.

I'd like to think the above statement false. I think something could be written to at least get people started. I'll try to collect the info here and if it gets unmanageable I'll put into my FAQ engine.

Here are the list of threads that I find relevant:

http://www.showeq.net/forums/showthread.php?t=3784&highlight=HOWTO
http://www.showeq.net/forums/showthread.php?t=3322&highlight=HOWTO
http://www.showeq.net/forums/showthread.php?t=3083&highlight=HOWTO
http://www.showeq.net/forums/showthread.php?t=5265

Outside the forum, I found these to be helpful:

http://samba.org/samba/docs/myths_about_samba.html
http://samba.org/ftp/tridge/misc/french_cafe.txt
http://us1.samba.org/samba/ftp/slides/net_analysis.pdf

Beside the standard tools sniffers (tcpdump, ethereal) these tools seem to have potential?

ftp://ftp.procplace.com/pub/tcl/sorted/net/sockspy-1.0/1.0
http://samba.org/ftp/unpacked/junkcode/udpspy.c

The above should give someone who is interested in helping some reading material. It helped me.

Now, I guess is the questions, most of these are what I think should be done.

Patch Day

Opcodes:
Q: How do you discover opcodes have changed?
A: Run seq and log into a quiet zone
Press F8 to log Unknown Data
Do something
<need help here?>

Q: Opcodes changed, how do I find the new opcodes?
A: Run seq Press F7 to log Zone Data
Log into a quiet zone
Do something
<need help here?>

Structures:
Q: How do you discover structs have changed?
Q: Structures changed, how do figure out the new structs?

Packets:
Q: How do I de-serialize a packet into a useful struct?
Q: Encrypted packets do I care?
Q: Huge udp packets and fragmentation, do I care?

[purple]

Personally, I see two ways people do opcodes right now. Both are pretty important to me and when when opcodes go into seq, both ways are used.

For Client->Server opcodes, people decompile the .exe and pull them. These people are way smarter than I am. I have no idea how this works. I just know that they fill in tons of opcodes much faster than it would take me doing the way I use.

The way I do opcodes is just to read a zone.log. When seq is working, turn on zone logging and leave it on for a day of playing. This will build up a reference for what packets look like. Then, when the opcodes change, use your reference to fix them.

Zone once, and compare the packets for that zone to your reference zoning. This will fill in about 20 of the most important opcodes for seq. By compare, I just mean read the zone.log with changed opcodes and compare it directly to the one with known opcodes. Most of the stuff stays in the same order and if the packets are the same-ish size with same-ish content, then the opcode can be easily found.

After this, just pick well known opcodes to fix and just cross reference between the reference zone logs and the new zone log to fill them in. I usually do map opcodes next (OP_NewSpawn, OP_Death, OP_ClientUpdate) and after that seq is working well enough to use for me. On my todo list is to add an importance field to the opcode XMLs so we can prioritize opcodes for seq, but I haven't done it yet. Once I do, it should be obvious which opcodes get the best bang for your finding buck.

Lastly, I walk the opcode list for ones not updated yet, produce them in the new zone log and cross reference them with old zone logs to make sure I have the right packet. This takes the longest and sometimes I'll release seq without having them all filled in because most of you don't care as long as the map and spawnlist works!

That's the whiz-bang process I go through. It's slow and annoying, but I'm used to enough packets now that I can remap most of the opcodes in a couple days. With the help of the people who usually help, this can get done in less than a day sometimes.

Structs are a bit harder. Once the opcode is correct, you'll start seeing size mismatch errors on the console if the struct has changed size. Alternately, if the struct is the same size, but different, you'll see garbage. Note that seq doesn't use a lot of the opcodes, so some struct changes leave us behind because we don't process a lot of the opcodes.

For structs, you just take the old everquest.h struct and try to overlay it on the new packet and see what is changed. Some people use perl scripts like structvis for this and that works well. I usually just pick an easy to identify part of the struct, find it in the packet, and compare offsets. If they are right, the change is further in. If they are wrong, the change is more towards the beginning of the packet.

If the struct is totally changed, sometimes you just have to throw away the whole old struct and re-find things. This is harder.

A lot of times, if all you care about is seq working, you just need to find where the added bytes were in the packet and increase an unknown there. If you want to increase understanding of the stream, try to figure out what the new data actually is.

For the most part, you shouldn't care about encrypted packets or fragmentation or combination. That's all taken care of at the protocol layer. If you do want to learn how it works, read the code and use the global log to see encrypted packets and the raw logging in zone log to see unencrypted but still rolled up combined packets.

Overall, you'll get a lot better at it as you do it and get used to what packets look like and can quickly do conversion to/from hex and float. Like anything, you get a feel for how things work and what kind of changes usually happen the more you do it. It's so not rocket science.

[BlueAdept]

Here are a couple of threads.

Remember if a sub link has seq.sourceforge.net at the begining, you need to change the link to www.showeq.net or the link wont work.

http://www.showeq.net/forums/showthread.php?t=4913

http://www.showeq.net/forums/showthread.php?t=4953

http://www.showeq.net/forums/showthread.php?t=4855&page=1&pp=15

The impl.pl file can be found here http://eqitems.13th-floor.org/misc/implen.pl.txt

As I was browsing, I also found this. http://eqitems.13th-floor.org/misc/optable.pl.txt
Both files might need updating. I don't know. I haven't used them.

Of couse I wish to thank ksmith for those files.

http://www.showeq.net/forums/showthread.php?t=3917

http://www.showeq.net/forums/showthread.php?t=3921
I think that is all for now.

[ksmith]

The implen.pl and optable.pl scripts are obsolete, but do provide inspiration for future disassembly scripts.

Currently structvis.pl is the most important script for patch-day updates. I know some of the Emu guys have been working on an automated opcode guesser, but I'm not sure how far along it is.

[tanner]

What is still missing from the links below is how to de-serialize a packet (stream of bits on the wire) to a C/C++ structure.

I might be missing it or it might be the 'stare at the code' analogies.

I'm staring at the ~19k OP_PlayerProfile

May 18 2005 15:30:33:780 [Decoded] [Server->Client [Size: 19560]
[OPCode: 0x1f1c]
[Name: OP_PlayerProfile][Updated: 05/11/05]
000 | bc ce c1 29

Skip the part that it is the checksum of the packet, how is it determined that it is a uint32_t?

When the package went from 18496 to 19560, I can see the addition bytes in the packet. I can also see that these new bytes fall at the 'right' location for the aa_array.

How was it determined that the bytes belonged to the aa_array and not something new, like a new entry in the struct?

Only way I can think of verifying this to have a packet before the increase in size, have -max- AA(!), log your session, go get your vet AA(s) and see what this new packet looks like (ie the staring at the data).

In general is that how structs are flushed out? I think I'm missing some fundamental concept. :-)

[purple]

Magic and guessing mostly.

For this last patch, I saw that maps didn't load right, which meant zoneId in charProfileStruct was wrong. So the change was before zoneId. So I traced it back using easily found fields like plat and servername until I found that servername was off by 960 and stuff before aa_array matched my old charProfileStruct from before the patch. Then I realized the patch message said something about people hitting the AA cap. So I put 2 and 2 together and assumed the doubled MAX_AA from 120 to 240. This fit the wire data, so it was good enough for me.

[blvan]

to any mods out there, can we get this post stickied? This is a nice place for a novice to start with helping to diagnose some of the opcode changes, so there are more then 3-5 people are helping the community after patches.


TIA.

[BlueAdept]

Good idea. I am going to move it to the development thread and sticky it.

[QuietWolfe]

Many, many, many, THANKS.

I have been learning and researching since TBS expansion on how to do this and help.

This will really help get some of the "more than novice" users of SEQ to be more helpful in determining opcodes and struct changes and finding even some of the more opaque ones that they might want that most users dont even look for.

Keep up the great work on this. I have played EQ for going on 8 years. It is still a great game even with all the other so called "advanced" MMO's out there. SEQ offers a really great tool, to help keep a great game alive. Too bad, Sony doesnt realize that with just a few, minor advancements in the existing game engine, that EQ would still be the #1 seller.

Again.. thanks to all that participate here and help us "noobs" understand where we can help too.