What is the methodology used to determine the packet structure?

[joojooga]

I would guess it all starts with a packet capture using something like Ethereal or Sniffer. But once you have the data, how would you go about determining where frames and headers begin and end? And what is the methodology used to decrypt the data in those if they are encrypted?

[Poncho]

I'm in the process of learning this myself as this same exact post seems to trickle out everytime SEQ is not working for some amount of time...I think it was Ratt that hit the nail on the head in one of his replies..(sorry too lazy to find the actual thread)...but it went something like this:

Most of the responses in a nutshell just say to goto an empty zone and start doing shit. You'll notice changes and tendancies in your data. This should, in theory help you find what you are looking for.

Unfortunately I'm not good at this (yet) and cant help more than that. I also realize that there is really no way to put into words how to teach someone how to do this....Just a matter of "time behind the line" so to speak.

Poncho

[fester]

Capture every packet.

Isolate the parts that change and the parts that stay the same.

If you determine that it is compressed or encrypted, look at the data a short while to see if it is clear what protocol encoded it. If it is not quickly clear, open a disassembler and look at the source code.

Once you know exactly how to decode the data, look at the unencoded data. Do the same action over and over again until you isolate struct values and their meanings. If your lazy, look at your disassembled output and obtain from it the size of all the elements of a struct (especially useful if there are very large structs like item and char profile structs.)

Really very easy, but can take time and considerable patience.

[joojooga]

Excellent. Thx both replies. I think I'll take a shot at it. I didn't even consider a most excellent point. Piddle around in game doing actions you *know* what the data will be.

YOu'll know your Loc, what spell you cast, what zone youre in, level etc...

[Alfred]

Originally posted by fester
Capture every packet.

If you determine that it is compressed or encrypted, look at the data a short while to see if it is clear what protocol encoded it. If it is not quickly clear, open a disassembler and look at the source code.

Once you know exactly how to decode the data, look at the unencoded data.

I've been a C and then java programmer for the 'professional' world for about 13 years and I think the above is a lot harder than it sounds. One would have to have intimate knowledge of the disassembler, the assembler language, and what diferent protocol/compressions look like. Not to mention that back when I was using a disassembler... not all disassemblers are created equal. Maybe that has changed...

[baelang]

SOE is using pretty standard libraries for things like compression.

[fester]

I made the assumption that the user had a rudimentary knowledge of disassembling, Intel assembly, and many "common" compression and encryption methods.

[SilentVoice]

Been looking at some of the code lately. Does anybody have a stub program that will decompress/unencrypt a specific and specified section of the captured data? If not, I'll look into writing my own. I just hate to waste time starting from scratch if someone already has it written.

[Amadeus]

You all realize that if someone wrote a decent HOWTO on this topic, we could greatly increase the number of available devs on SEQ.

Personally, I feel pretty comfortable manipluating the code, and doing my own custom changes (adding features as well) to SEQ...however, I'm utterly clueless on how you guys determine packet structure and opcodes, in all honesty.

[fester]

You can not write a HOWTO on this task (at least I would not know where to start.)

It is like using a handgun. Some people are natural and some people have to work rather hard to be a good shot.

All this is based off hunches. You look at the data and you just know what it looks like. If you do not know, then you have to spend a bunch of hours looking at the source.

[Amadeus]

Yes...but writing a bit about the exact software, decryption software, etc... and perhaps just a simple example of one structure found (screenshots) ...would be excellent.

[joojooga]

Hey fester,

I have no knowledge of any of that stuff. I do know that one can achieve a working knowledge of just about anything by studying and understanding the overall concepts and methodologies of a particular field.

That having been said, I've been forging ahead with my quest at about 15 minutes a day with C++ as my starting point . I purchased a book called Ivor Horton's Beginning C++ by Wrox. He really drills down on the boring side of C++ but it's the fundamentals that I'd like to understand firmly. Another book that I work alongside that simultaneously is called Teach yourself visual C++ in 21 days. Radically different book. The fundamentals of the one do however "snap into" the second one.

I took a screenshot of a typical ethereal capture of Everquest traffic and put it up here. (http://www.freewebz.com/cpt_cam/images/capture1.jpg) This is typical of EQ traffic. UDP back and forth. I believe all the information you need is in the DATA portion of the UDP packet. Now that I see the packet's data I ask could myself a million questions..

1. What is it encrypted with?
2. Is it compressed? How do I know if it's compressed?
3. What am I looking at? Was that me conning something or zoning, etc?

I already have a background in networking so I have an advantage in that area. I chose my weak side (programming) for study so that once I break the information out of the packets I can use the programming side to take that information and make something meaningful. (A mob list, a map, levels..etc..) But how does one "break the information out of a packet?"...Perhaps in my study of C++, I will learn that maybe it's easier for me to pull the data out of memory instead of decoding packets and send that on..who knows...

As you can see, for those interested in this pursuit, there's a lot of work to be done to catch up to the expertise of the developers on this board. I agree with Amadaus that there needs to be a thread that relates the general outline of what to do, and the tools used in each step. I also agree with fester that you could never write a full HOWTO on this. But we CAN develop a HOWTO on the process and leave the details to the people learning, because when the student is ready, the teacher will arrive.

As for those that are already there, I am truly impressed. I just hope that the fundamental building blocks I am learning now get me to where you are in this lifetime.

-Joojooga

[high_jeeves]

I think you guys are overcomplicating the issue. You start a capture program, you do things in game.. you look at the packets as you do things.. (sit and start over and over in an empty zone, and watch the packets).. do this with a variety of actions, look for patters, identify those patterns.. As for compression and encryption: disassemble the eqgame.exe for information on these... i use w32dasm to do this, but there are a variety of others that will do the job just fine.. like fester said, there isnt really a "right way", we just look at the disassembly, look at the packet stream, and spend time figuring it out..

Also, read the ShowEQ source.. although it doesnt work right now, it will give you an idea of some basic structures (since they have changed, they arent 100% accurate), and some of the basic compression information..

--Jeeves

[Amadeus]

Hmm...thanks Jeeves for a great posting! Personally, I know how to find offsets in eqgame.exe from my workings with MQ. And, I suppose if I had some "roadmaps" from before, I could probably find offsets for ShowEQ using the same method (ie, looking for familiar packets, etc..).

However, no one really ever explained how one determines a structure with MQ...only how to find offsets.

[high_jeeves]

You just look at it, and figure it out.. there is no magic formula here... look at all of the spawn structures, look for similarities and differences.. figure out why a byte changes, and for what reasons... thats it.. its just pure common sense, there is no methodology..

--Jeeves

[joojooga]

Thx jeeves.

[tamasine]

just for kicks, and based on that fruity concept of open-source having multiple inputs.. could someone post the current code-fragment that de-mangles the packets prior to op-code identification/decompression/decryption?
This bit is not easily done with packet stream analysis (as commented by jeeves) but by eqgame.exe disassembly. I would love to help with identifying stuff - i was about to add bazaar packet structs/code before the last change.. but many years ago i focussed on sensible cpus (mainly ARM) after tangling with a z80. x86 has always been a black art to me that i have neither time nor inclination to try to comprehend. i salute those who do understand it, but it's not my bag.

it seems to me that at the moment everyone is doing the same thing, and indeed being told to replicate effort. the atomic doesn't-work => fully working update cycle of the cvs tree currently also seems like something that others are disuaded from contributing to - if only because people who can contribute to the later stages of the cycle (pinning down structs et al) simply can't help.

tam.

[edit. okay, i went away for a week, and this subject got raised elsewhere while i wasn't looking. the code fragment request stands, though i still expect inc. flames for the opinions]
yes, go on. flame me. it's an opinion i hold.. not an absolute.

[Fantastik]

Ok here is the deal. What we NEED are a suite of tools to manipulate the data stream outside of seq.

I'm all for writing and maintaining the tools. A few problems though. The decryption/compression libs (from what I understand) are closed source for a reason. We dont want these libs public knowledge or you'd have a seq windows port persumably.

I think this is a BS excuse because, as others have stated, its easier to either rip the structs directly from eq memory (windows box - High-Jeeves) or simply dissemble and recode the libs if you are bent on doing it. Its been sugested by more than one person that the only way to play with the structs is infact to disemble and decode them yourself... exactly what the closed source stuff is already doing. Why are we telling people on one hand not to mess with this (the closed source libs) but on teh other hand telling them to replicate the work of the closed source libs themselvs to decode the stream. Its nonsensical.

Ok I'm rambling. Basicly I find fault with the whole idea of the encryption and compression being closed. The port is going to happen anyways. And, honestly, SOE deserves it for mucking with the protocol stream simply to obfuscate things.

If we had some sort of open lib going, we could have 2 teams of people. 1 team working on fixing the decode/decompress algos right after a change, and a second team that works on the struct changes after team one finishes thier job. This is ideal seeing that most people are uncomfortable with point one, but wouldnt mind helping with the tedium of point 2.

I havent mussed with asm. since risc vlsi classes as an undergrad. I vaguely remember looking at x86 asm and throwing up. I'm confident i could dissemble both the closed source lib, and SOE's product to come up with the algos for decryption and compression given enough time... most threads I've read seem to allude to the fact is a simble zlib compression and xor, bit rotate encyption.

The question is, why do I have to do this? Can someone thats already done this simply post the darn stub or am I gonna have to redo work that multiple people have already done?

Edit: Also. I'm lazy. (Most programmers are) And the thought of having to look at x86 asm code makes me nauseous. I don't want to have to resort to it. Well, it'd be inevitable when the enc changes, but from what i understand the enc/dec hasn't changed in a while, just the underlying struct stream.

[high_jeeves]

Well, since the closed source libEQ.a no longer appears to be valid, its a mute point... Again, if you want it done, start working on it.. or atleast got o IRC, and talk to the devs about it... Finding the opcodes and structures is the relatively easy part (compared to solving the new packet compression/encryption/encoding scheme)...

--Jeeves

[Fantastik]

I intend to start working on it. I'm trying to get a grasp of the situation before i duplicate effort. I'll try the suggestion of the irc channel. I need more info of what has actually changed. Seems to be alot of misinformation going around. I thought the structs for the opcodes were changed, and new opcodes were added. You are telling me theat the encryption/compression was changed also.

If true it does indeed make it more of a pain in the ass.