Please...pretty please

[Amadeus]

Please, with sugar on top, could someone take the time (/point codepig) to write a some sort of HOWTO/GUIDE on how they discover opcodes and/or structs after they change?

Someone FINALLY did this on the MQ project and it was like a dream world opened up. I doubt there will ever be a point without structs there again because people were educated about it.

It would just be so nice to be able to contribute. Trust me, there are a lot like me that are really close, but simply have not been exposed to the mysteries of sniffing packets, decoding them, and then discovering the offsets and structures.

Thanks

[Alfred]

I didn't use the search, but didn't someone just post within the last week or two a thread about this?

It was a pretty good discussion. Had sample source to compile and advice on where to look in memory. If you've never touched a C program then I would guess it would be a little more difficult. Your milage will vary. ;)

[grimjack]

Originally posted by Alfred
I didn't use the search, but didn't someone just post within the last week or two a thread about this?


I think I remember that thread. However with that said, codepig came through with new structures and I'm sure Amadeus would like to hear what was done to find it from the source (I can't say I wouldn't want to hear it myself). Amadeus asked a similar question at the MQ boards and after he got the correct answers he really has put out some great code for MQ.

Also, if this is the Thread (http://seq.sourceforge.net/forums/showthread.php?s=&threadid=3432) you where talking about Amadeus was in that discusion.

Thanks

[Wildcat]

I'm not sure of a HOWTO but when I needed to find BeginCastCode, I turned on 'View Unknown Data' off of Network, went into the game and spam casted a spell and noticed that opcode 0017 kept coming up.

Copy the line that it gave for the payload of the opcode into an editor, checked to see what it reported my player's spawnID was, and that gave me 4 bytes out of the 10 that make up the data. Checking spells.h told me what the code for the spell I was casting was, and that gave me enough to update everquest.h for the new structure.

Now, this opcode was reaptable on demand so was probably a bit easy in that regards to figure out, but many of the other broken opcodes could be done in a similiar fashion.

[Amadeus]

Even if somene posted a very ugly walkthrough and I could decypher it with questions back and forth (as was done on MQ boards), I could re-write it for general consumption if were desired.

I know MQ offset/struct finding is utterly different and probably easier. But, that still doesn't mean it can't be taught :)

[fester]

Rough Draft (little of no forethought):

Prep work:
1) Have a Diff command (I do my work from the seq box) to compare
hex dumps.
2) Make sure opcode logging is working in showeq and the disk has space.
3) Log ALL opcodes with hex dump info.

This should isolate virtually all "repeatable" opcodes
a) Go to a quiet zone (Arena near Lake Rathe).
b) Repeat the action (10-20) times until you see an opcode
that keeps repeating and appears to be more common when you do your action.
c) Focus on the most likely, then make sure it does NOT appear when you
are not doing your action.

Please reply with an example of a non-repeatable opcode, I spent 5 minutes trying to think of one.
I don't believe there are any real opcodes which can not be repeated.

Ok, now you have all opcodes done.

This should work for all data in structures:
a) Look for the hex value of your known data (Level, SpellID, PlayerID, etc.)
Say level 65 in hex is 0x41, look for "41" in hex dumps. Keep in mind byte order and big/little endian format.

Example (Fake Opcode):
00000000 41 00 00 00 34 12 00 00 09 00 00 00 |A..."..........|
Level: 65 SpellID = 0x1234 (4660)

Notice 41 (level 65) is the first byte. It is followed by 3 0x00 bytes. This LIKELY means
it is a DWORD (or 32 bit value). Spell ID 4660 is 1234 in hex and we find "34 12" in the structure. Notice the value has intel byte ordering. It is also followed by 2 0x00 bytes, also likely 32 bits. We also have this "unknown" data. 0x9, which we realize is the level of the spell we had casted (a level 9 spell.) So we know that the 3rd 32 bit value appears to be "spell level".

So we craft this as:
struct ubercast
{
/*0000*/ uint32_t level; // Character level
/*0004*/ uint32_t spellid; // Spell ID
/*0008*/ uint32_t spelllvl; // Spell level
};

b) You must do this and must ASSUME showeq structures are WRONG. In other words, do no overlook examining the area of a structure that showeq believes is SpellID if you need to locate Level and it is not showing up in any of the "unknown" areas of the structure. Believe it or not, often you will find structures shifted up or down a certain number of bytes and this will make elements contain data that belong in other elements.

Example a change from the above struct:
00000000 09 00 00 00 64 00 00 00 34 12 00 00 41 00 00 00 |....d......A...|
Our level is still 65, SpellID still 0x1234, Spell level still 9.

Notice Character level was move, so was spell level (actually they were swapped), and "64 00 00 00" was inserted. We are curious on the 0x64, and it appears to be 100 (decimal) which is how much mana the spell takes to cast. So the new struct is:

struct ubercast
{
/*0000*/ uint32_t spelllvl; // Spell level
/*0004*/ uint32_t manacost; // Spell Mana cost
/*0008*/ uint32_t spellid; // Spell ID
/*0012*/ uint32_t level; // Character level
};

c) Say you know what you want to find, but do not know where to look or how it may be stored. Say you are looking for AAExp information. You know the bar and you know what it looks like, but you do not know how it is stored in memory. Well first make an assumption that it is stored in the "CharProfile" packet. Do as little as possible while changing your AAexp value. This is best done if you on one account prepare a mob for exp kill so that your showeq character will get full exp (no group) for the kill and there is very little you need to do to kill it (one nuke). So log on your showeq character, save his CharProfile structure, kill the mob, /sit, /camp, log back in the character, save his CharProfile structure in another file and diff the files.

Example:
1c1
< 000004f0 68 65 6c 6c 6f a5 00 0a |hello%..|
---
> 000004f0 68 65 6c 6c 6f a6 00 0a |hello&..|

Notice only one byte changed. From a5 to a6. Well a5 is 165 decimal. We "learn" that AAexp is at this position and we learn that the scale is 0 to 330. We first speculate the range because our AAexp is about half full (165 is half of 330) and we test this by finding a character VERY close to AA Ding and save CharProfile until he dings. We will see the highest value before (we notice it was 329 decimal) and the lowest value after (in this case 0 decimal.)

d) Sometimes you may be thrown off by "changing" values. Make sure you do not jump to conclusion if you see a change. SOE often uses "random" elements in places (Like CharProfile and ItemStruct to name a few.) These often appear to be constantly counting upward and should be excluded.

Ok, this concludes this message. Ask questions or praise/slam the parts you wish.
I am not sure this is useful, but there has been many that have asked for a “HOWTO guide on finding opcodes and structures” document.

[Amadeus]

This is superb...and EXACTLY what I am looking for! Let me try a bit of it on something somewhat unknown, and I'll come back for clarification.

One question before I begin:

* I am accustomed to working with structs over on MQ using VS.net data types. Could someone convert the following types to Linux/gcc (or whatever term I should be using)? ...thanks!

BYTE (1 byte)
WORD (2 bytes)
DWORD (4 bytes)
FLOAT (4 bytes)
CHAR


...I could probably go through everquest.h and guess, but confirmation would be nice.

[Amadeus]

Oh yes, one more question before I begin:

This tactic must be done using SEQ 'logging' correct? (because we need to have the packets decoded before exploring them)

So, tcpdump really doesn't help as much because the packets are encoded?

If someone could elaborate on this, that would also be helpful.

[Amadeus]

2) Make sure opcode logging is working in showeq and the disk has space.
3) Log ALL opcodes with hex dump info.


I'm sorry to already be asking a ton of questions; however, they should be easy to answer.

Anyway, by "log ALL opcodes" ..do you mean, "Log ALL packets"? I have options for "opcode monitoring" and "log packets" ....

[fester]

Originally posted by Amadeus
BYTE (1 byte)
WORD (2 bytes)
DWORD (4 bytes)
FLOAT (4 bytes)
CHAR


BYTE = uint8_t (unsigned 8 bits) or int8_t (signed 8 bits)
WORD = uint16_t (unsigned 16 bits) or int16_t
DWORD = uint32_t or int32_t
FLOAT = float (32 bits IEEE) or double (64 bits IEEE)
CHAR = uint8_t (unsigned 8 bits)

Generally 64 bits is "long long", uint64_t or int64_t.

Examine /usr/include/stdint.h or /usr/include/align.h for more info.

If you have a need to pack items without standard alignment, add a ":" after the item and put the size in bits after the ":" symbol. E.g. uint8_t hello : 6; for a 6 bit field.

[Amadeus]

strangely enough, I'm now completely clear on how to determine structures once the opcode is set.

However, I'm still a bit unclear on how to find the opcode in the first place. For example, Charprofile: I turned on 'log everything' and zoned a few times...then examined the log file and did not find my character name anywhere. However, when I went to opcode monitoring and entered the opcode given in opcodes.h...it logged to file beautifully and there was my character name in lights.

Does the simple 'logging' not decrypt? If not, how do you use opcode monitoring to look at all packets? (it doesn't seem to want to take a wildcard).

[fester]

Originally posted by Amadeus
Oh yes, one more question before I begin:

This tactic must be done using SEQ 'logging' correct? (because we need to have the packets decoded before exploring them)

So, tcpdump really doesn't help as much because the packets are encoded?

If someone could elaborate on this, that would also be helpful.

Well, this has multiple answers.

AFAIK showeq opcode monitoring only allows for 15 specified opcodes to be monitored (PLEASE correct me if wrong.)

Global logging (log all packets) does not strip the headers.

The headers in EQ packets are variable lengths, so using tcpdump is frustrating.

This is what I do. It may not be for everyone:

I add this code to packet.cpp (yes it is sloppy):
// fester debug printer (pretty)
QString myxxdit(char *data, int len)
{
QString tempstr, hexstr;
int j;

if (len<0)
return "Invalid Length passed to myxxdit\n";

tempstr = "";
for (j=0; j<len; j++) {
if ((j % 32) == 0) {
hexstr = QString::number(j, 16);
if (j!=0)
tempstr += "\n";
tempstr += hexstr.rightJustify(7, '0') + ":";
}
if ((j % 4)==0)
tempstr += " ";

hexstr = QString::number((unsigned char)data[j], 16);
tempstr += hexstr.rightJustify(2, '0');

if (((j % 32) == 31) || (j == (len-1))) {
int i;

if ((j % 32) != 31) {
for (i=j+1; i<=(j+32-(j % 32)); i++) {
if (((i % 4) == 0) && (i % 32))
tempstr += " ";
if (i % 32)
tempstr += " ";
}
}
tempstr += " ";
for (i=j-(j % 32); i<=j; i++)
if (isprint(data[i]))
tempstr += (char) data[i];
else if (data[i])
tempstr += "+"; // not zero
else
tempstr += "."; // zero
}
}
tempstr += "\n";
return tempstr;
}

Then go inside EQPacket::dispatchZoneData and add this line to the section checking CRC (line 1856):

Change

for (unsigned int packetNum = 0; packetNum < pktlist.size(); packetNum++) {
data = pktlist[packetNum].data;
len = pktlist[packetNum].len;
uint16_t opCode = *((uint16_t *)data);

switch (opCode)

To

for (unsigned int packetNum = 0; packetNum < pktlist.size(); packetNum++) {
data = pktlist[packetNum].data;
len = pktlist[packetNum].len;
uint16_t opCode = *((uint16_t *)data);
fprintf(stderr, "%s\n", myxxdit((char*)data, len).ascii()); // fester debug
switch (opCode)

This is the lazy man way. Also, worldzone packets should be on via "log world packets".

There is VERY likely a better way to do this, but I never bothered to look up how the "better" way was done, as this took only a couple minutes to implement and was a "sure thing".

[fester]

Originally posted by Amadeus
However, I'm still a bit unclear on how to find the opcode in the first place. For example, Charprofile: I turned on 'log everything' and zoned a few times...then examined the log file and did not find my character name anywhere. However, when I went to opcode monitoring and entered the opcode given in opcodes.h...it logged to file beautifully and there was my character name in lights.

Does the simple 'logging' not decrypt? If not, how do you use opcode monitoring to look at all packets? (it doesn't seem to want to take a wildcard).

Ok with the patch and sub I added you can see all packets decompressed, unencrypted, and separated by opcode. The packet is dumped to stderr with the opcode as the first two bytes in Intel byte order.

E.g. This opcode would look like the following hex:
#define PlayerPosCode 0x0022
0000000: 2200a608 ce238410 00208200 bde50700 00200300 ".+++#++. +.+++..

Notice 2200 is the opcode 0x0022.

There is a "right" way to do this inside Showeq and I am lazy. Maybe someone can explain the right way to have showeq dump a hex dump for every opcode separately (decompressed and unencrypted) to a log file?

To answer your question "simple 'logging' not decrypt", as I understand it dumps each PACKET. A packet may be encrypted, compressed and "packed" (when more than one opcode is included in a single packet or more than one packet of the same opcode is included in a single packet.)

[Amadeus]

Congratulations..you did it. It's a working guide on finding offsets/structures that a person can understand. The patch for packet.cpp that you provided is the real key, for me at least, in the puzzle.

So, let me see if I have this right...for the next time SEQ breaks (assuming encryption did not change)

1. create decode.cpp using Codepig666's great little app for Windows. Replace the opcode in that project by exploring eqgame.exe or a memory dump (I would use windbg.exe).

2. Compile SEQ. Use Fester's patch, and start seq in terminal window with output to a file to catch packets. The first two numbers of each packet is the opcode reversed (ie, 2902 = 0x0229)

3. Structures are actually the easiest once other steps are done. Use opcode monitoring and put in the opcode of the struct you want...save, and then start the process of logical deduction. (been there/done that with MQ).


Well....see, wasn't so hard :) Now, I just have to hope they do'nt change the procedure so much that Codepig666's little app doesn't work anymore ;)

I also assume that libeq.cpp/.h shouldn't have to be modifyed very often (unless something major changes) ...is that correct?

[fester]

Originally posted by Amadeus
Well....see, wasn't so hard :) Now, I just have to hope they don't change the procedure so much that Codepig666's little app doesn't work anymore ;)

I also assume that libeq.cpp/.h shouldn't have to be modified very often (unless something major changes) ...is that correct?

There is another thread I started about how to find the decode.cpp even when it is modified. There was a program written by usr387 to dump the decode.cpp file using a memory offset. The thread contains instructions on how to find the memory offset when eqgame.exe changed. So unless they remove this table or deliberately try to obfuscate the code, it will always be rather easy to find (like the key was easy to find when we needed a memory sniffer for the encryption keys.)

http://seq.sourceforge.net/forums/showthread.php?s=&threadid=3432

Libeq.cpp/.h needs to change whenever the entire method of packing and delivering packets is changed. To my knowledge this has happened only a few times in the history of showeq.