Guide to finding opcodes, does anyone have a good one (want to help "fix" Seq)

[Frank25]

I've long been wanting to help in the "Fixing" of SEQ after patches.
But i have no idea where to begin or what to do.
I'm fluent in installing and using linux but i'm not a programmer.
I do love SeQ and appreciate the work all the devs put in and want to help out doing the menial grunt work of repairing after patches.

Any help is greatly appreciated

[BlueAdept]

There had been a wealth of information posted about it, but unforunately the search only works from when the board was moved. You can use the old search for the old board http://seq.sourceforge.net/forums/search.php?s=

Im slowly going over the old threads. Ill post anything that find that might relate to finding op codes here.

From Baelang


* go to the src directory
* use our old pal "grep" to find the files that contain the text you are interested in. for example: grep "define PlayerPosCode" *
* next view the file and check out how they are used to kind of get an idea what to do. for example: vi -R opcodes.h
* make a backup copy of the file you are about to change, then edit the file as you see fit. recompile and test.

Third, ask any followup questions in the format:
* I am trying to do XXX
* I (searched and) found YYY
* I tried ZZZ and it resulted in AAA
* The bit i am stuck on is BBB
* could you please suggest where i go from here?
this shows that you are working on it dilligently, rather than just want someone to hand you the answer. you will get much better responces that way.

Forth post a followup once you find your answer so that someone searching on the same problem will find their answer too.

at least thats what works for me.


From Fester


A good place to start is with the Network->Log->All function. It will log all packets to the default file /usr/local/share/showeq/global.log This file contains the UDP payload of all packets to or from the EQ client. Its in a pretty difficult to read form. Spend some time with this data to get a feel for what it is you might be interested in.


I have a handful of utils I keep around for parsing these packets out. But more often than not I look at the data in hex. After a while you'll get a feel for it and things will start to become obvious.

Fee


From iluvseq


I know showeq has an option to log invalid opcodes (ie: packets that come in with an opcode that doesn't match one in opcodes.h)

Also, if you witness it giving you errors regarding struct size mismatches, this generally means that SOE has changed a structure a bit (or that the opcode has changed) either way, you can set showeq to log packets that match a specific opcode. Then you can analise those packets, to determine if (a) they represent a different packet (opcode changed) or (b) they are just a tad off from the expected packet, but with different structure (struct change)

Once you've narrowed it down, you can start fiddling bits in everquest.h and re-compiling till you've figured out the new structure format.

Hope that helps some.


Even though this isnt quite about opcodes, it might be relevant in the future
From Mvern


Its kind of funny how quickly people jump to conclusions, especialy without knowing what all is involved in fixing SEQ after an encryption or netcode change. Its not just 'fixing a few opcodes', its breaking out a disassembler, hunting down the decryption routines, following whats happening with a debugger, logging a large amount of packets, and much trial and error, just to regain basic functionality. On top of that, things were changed around on test again a few days ago, including the encryption and some major struct changes.

Personally, I've been focused on deciphering whats changed on test since spending time fixing things for live, only to have it patched again in a couple of days is a bit of a waste.

mvern


By Fee


Having not looked at the changes, I can hazard a guess as why it is not fixed.

In the last state, there was a table inside eqgame.exe containing 4 byte words for every opcode. This table contains either 0xffffffff or a size in bytes.

Click link for a list of the last known sizes (items not listed as 0xffff):
http://cvs.sourceforge.net/cgi-bin/....viewcvs-markup

If the information ShowEq is using is wrong, then ALL FLAG_COMBINED packets will be corrupted (because we jump to the wrong position) and almost half the packets will be missed. The other half could also be invalidated if their structure was changed.

I have maintained a private ShowEQ since Luclin. I have not examined the changes because my system with the tools (IDAPro, MS VS6.0, etc.) suffered a hard disk failure in late March and I haven't bothered to reinstall that system. I also would like to see non ShowEq tools get into wide spread use. This may make it less interesting to change things that break ShowEQ.

To find this location inside eqgame.exe, disassemble and look for "and e.., 00000FFF" in the file. This will be rare. For example in the last disassemble I had around, this was the list:

Valid:
:00554BFD 25FF0F0000 and eax, 00000FFF
:00554CA4 25FF0F0000 and eax, 00000FFF
:005551BD 81E6FF0F0000 and esi, 00000FFF
:0055529C 25FF0F0000 and eax, 00000FFF

Invalid:
:005B7E65 25FF0F0000 and eax, 00000FFF
:005D86BA 81E1FF0F0000 and ecx, 00000FFF
:005DBB56 81E6FF0F0000 and esi, 00000FFF

Look at the VALID ones and ignore the INVALID ones. How do I know the difference? Well valid ones have code like this following the AND instruction:

:00554BFD 25FF0F0000 and eax, 00000FFF
:00554C02 897E0C mov dword ptr [esi+0C], edi
:00554C05 668906 mov word ptr [esi], ax
:00554C08 8D5E08 lea ebx, dword ptr [esi+08]
:00554C0B 0FB7C0 movzx eax, ax
:00554C0E 8B048574218400 mov eax, dword ptr [4*eax+00842174]

Notice eax is used as a index into 00842174 (a 32 bit word since the index is multiplied by 4.)

All one needs to do to get the table is to dump the data at 00842174 with a memory snooper. This table is build during eqgame.exe startup, so it is not trivial to examine a binary to obtain the table. Look for something like this:

:004F6BBF B9FF030000 mov ecx, 000003FF
:004F6BC4 83C8FF or eax, FFFFFFFF
:004F6BC7 BF74218400 mov edi, 00842174

This is somewhat rare (605 "mov edi, 0.*") and many dups (297 uniq "mov edi, 0.*" references) with many of them low (232 below rsrc (Imagebase = 00400000h + Object04: .rsrc Offset: 0023F000 + Size: 00001000 or 00640000h) leaves you with only 65 remaining targets. Very few (only 42) of the pair (or eax, FFFFFFFF followed by mov edi, 00640000 and up) appear.

Someone could write a program to disassemble, sort for these 42 events, them memory dump (from a running eqgame.exe) those 42 locations looking for what resembles the table (large chunk of mostly 0xffffffff values.) This could be used to automate finding the new opcode sizes after a patch. I am too lazy to do so.

This is the only "hard" part of fixing opcodes. Fixing structures is simple (once you have correct opcodes parsing.)

This post may please the people who complain "no one provides locate opcodes and structures lessons," but I still maintain that classes of this type can not make you the next Mvern. They are all self taught.

(BTW this DOES NOT disagree with Ratt's "Opcode changes ... main problem right now is the packet structure changes." It clarifies what he likely means. The structure of the packets to large degree is dictated by this opcode size table. Incorrect sizes for the "packed" opcodes will mean you will not find any of these very common opcodes.)

Archaeopteryx, opcodes used to be a few hours when you ran ShowEQ and watched unknown opcodes until you located the ones you need. This is no longer possible. You must disassemble and read process memory to be able to get to the point of "watching unknown opcodes to pin them down" now. This does only take a few hours, but instead of being able to be done in parallel by many people (perhaps maybe 50 or more), you are down to the few who know how to program in C (to read process memory) and who know how to read x86 assembly (to know where to look in the first place.)


This is probably one of the better threads on it:
http://www.showeq.net/forums/showthread.php?t=3432

This is also a fairly good thread:
http://seq.sourceforge.net/forums/showthread.php?s=&threadid=3322

Still a ton of stuff to go, but I have run out of time. I welcome anyone else to plug in some usefull information. I was going page by page, but this search turned up most of these gems.

http://seq.sourceforge.net/forums/search.php?s=&action=showresults&searchid=121609&sortby=lastpost&sortorder=descending

Edit: Made a mistake, one of the quotes was from Fester not Fee.

[ksmith]

In the last state, there was a table inside eqgame.exe containing 4 byte words for every opcode. This table contains either 0xffffffff or a size in bytes...

What Fee is talking about in that post is the implicit length opcode table. My implen.pl script does all the hard work of deciphering that.

If someone were so inclined as to trace through the opcode dispatch code, you'd want to look for "add eax, 0fffffff5h" to find the approximate location of the first opcode dispatch function (there are two). The code around there looks like this:



447b23 !
...... ! loc_447b23: ;xref j447ad2
...... ! push esi
447b24 ! mov esi, [ebp+10h]
447b27 ! ; eax -= 11
...... ! add eax, 0fffffff5h
447b2a ! xor ebx, ebx
447b2c ! ; is eax > max opcode?
...... ! cmp eax, 366h
447b31 ! push edi
447b32 ! ja loc_opcode_dispatch2
447b38 ! ; convert the network opcode to an internal opcode via lookup table
...... ! movzx eax, byte ptr [eax+data_opcode_table1]
447b3f ! ; jump to the code for handling the internal opcode
...... ! jmp dword ptr [eax*4+data_opcode_addr_table]


The second opcode dispatcher doesn't use a lookup table:



44a3f6 ! ; you'll end up here if the opcode was greather than the max opcode
...... ! ; or if the opcode translated to 0xbc
...... ! loc_opcode_dispatch2: << show xrefs (185) >>
...... ! mov ecx, [?data_82feec]
44a3fc ! call sub_40d6b4
44a401 ! test eax, eax
44a403 ! jz loc_44bbd4
44a409 ! ; edx = network opcode
...... ! mov edx, [ebp+0ch]
44a40c ! mov eax, 17fh
44a411 ! cmp edx, eax
44a413 ! ja loc_44b282
44a419 ! jz loc_44b273
44a41f ! mov eax, 0e2h
44a424 ! cmp edx, eax
44a426 ! ja loc_44a97f
44a42c ! jz loc_44a974
44a432 ! cmp edx, 65h
44a435 ! ja loc_44a605
44a43b ! jz loc_44a5fa


The real problem comes when you're trying to figure out what the code for each opcode does. For example, opcode 0x004f:



...... ! loc_44a53d: ;xref j44a516
...... ! push ebx
44a53e ! push ebx
44a53f ! push ebx
44a540 ! push ebx
44a541 ! push ebx
44a542 ! push ebx
44a543 ! push ebx
44a544 ! push ebx
44a545 ! push esi
44a546 ! ; 1459 The door says, '%1'.
...... ! push 5b3h
44a54b !
...... ! loc_44a54b: ;xref j44a6c2
...... ! lea eax, [ebp-30ch]
44a551 ! push eax
44a552 ! call sub_get_eqstr_sprintf
44a557 ! add esp, 2ch
44a55a ! push 1
44a55c ! push 0fh
44a55e ! jmp loc_44a969


Personally, I've never had a door talk to me but the code is there if one ever decides it has something to say. But that was an easy one, it called one of the eqstr functions (they grab the strings from eqstr_us.txt). Not all opcodes do... in fact, most of them don't use eqstr functions because they don't have to write any text to your chatwindow.

Edit: fixed formatting in the code blocks

[ksmith]

Baelang, Fee's first quote, and iluvseq describe the way figuring out opcode and structure changes are normally done, and I would reccommend that anyone interested in fixing showeq should use that method.

[Frank25]

Thank you guys for puttin me on the road to smartness heh.

Like i said i'm not a programmer by far but i'll be pooring over these tips and trying to see if i can figure out how you guys do this.
I'm very good at problemsolving most linux based code/programs so far so hopefully i can reverse engineer your genius :P

I'll be sure to post any progress i've made and more specific questions if i have any.

Thanks though for the very informative reply