Newby
11-27-2015, 03:55 PM
Ok, starting this thread so we can share info on what needs to be done on patch days. I'm still paranoid about including any information that might identify me, and I recommend that everyone else do the same.
I'm fluent in C and C++ and I'm comfortable messing with showeq's source code to get it working after a patch. However, there's a lot of code there and I haven't absorbed enough of it to consider myself a developer. I once tried to fix the bug where it doesn't show your position and heading correctly when you get on a boat and wasn't successful. The variables involved were all of different sizes and units and it got kind of nasty.
I used to do everything by hand and it would take a couple of days to get it working. But I've slowly been scripting things and that speeds up the process a lot. Most of my scripts are in Python.
I have a set of opcodes I identify. When I first started doing this, I wanted to keep the list down to the minimum necessary to get showeq working again. For a manual process, that makes sense, the fewer you need to identify, the quicker you finish. However, if you've got a script to do the job, then time becomes much less of a factor. The more opcodes you identify, the easier it is to find other opcodes, using the process of elimination. With the smaller packets, 8-bytes and less, there's so little data to look at. The process of elimination may be all you have to work with.
So this is my current list of what opcodes I look for:
OP_AAExpUpdate
OP_Action
OP_Action2
OP_ClickObject
OP_ClientUpdate
OP_CommonMessage
OP_Consider
OP_Death
OP_DeleteSpawn
OP_ExpandedGuildInfo
OP_Find
OP_FormattedMessage
OP_GroundSpawn
OP_GuildMOTD
OP_GuildMemberList
OP_GuildMemberUpdate
OP_MobUpdate
OP_NewZone
OP_NpcMoveUpdate
OP_PlayerProfile
OP_RemoveSpawn
OP_SendZonePoints
OP_SimpleMessage
OP_SpawnAppearance
OP_SpawnDoor
OP_SpecialMesg
OP_TargetMouse
OP_TimeOfDay
OP_WearChange
OP_ZoneChange
OP_ZoneEntry
On patch day, I started showeq and get it to log zone data. Then I get in the game and do a very specific set of actions that's guaranteed to generate at least one of all of the above opcodes. Then I turn logging off and start digging through the data.
I do this to get a timestamp on the zone.log file:
mv zone.log `date +zone.log.%Y%m%d`
and then I run the following to break up the log file into multiple files, one packet per file:
#!/usr/bin/env python
import sys
import os
def open_outf (work_dir, outfn_n):
outfn = work_dir + '/xx' + ('%06d' % outfn_n)
outfn_n += 1
outf = open (outfn, 'wt')
return (outf, outfn, outfn_n)
if len (sys.argv) != 2:
print 'usage: zone-log-breakout.py <zone-log>'
sys.exit (1)
infn = sys.argv[1]
(_, _, ts) = infn.rpartition ('.')
work_dir = 'work-' + ts
os.mkdir (work_dir)
outfn_n = 0
opcode = ''
(outf, outfn, outfn_n) = open_outf (work_dir, outfn_n)
inf = open (infn, 'rt')
for line in inf:
line = line.rstrip ()
if line == '':
outf.close ()
if opcode != '':
os.rename (outfn, outfn + '.' + opcode)
opcode = ''
(outf, outfn, outfn_n) = open_outf (work_dir, outfn_n)
else:
outf.write (line + '\n')
if line.startswith ('[OPCode: '):
opcode = line.split (' ')[1].rstrip (']')[2:]
outf.close ()
inf.close ()
That gives me a directory named "work-<timestamp>" with all the packets in the log file in separate files named "xx<seqno>.<opcode>".
That's enough to start, we can discuss identifying particular opcodes in other messages.
I'm fluent in C and C++ and I'm comfortable messing with showeq's source code to get it working after a patch. However, there's a lot of code there and I haven't absorbed enough of it to consider myself a developer. I once tried to fix the bug where it doesn't show your position and heading correctly when you get on a boat and wasn't successful. The variables involved were all of different sizes and units and it got kind of nasty.
I used to do everything by hand and it would take a couple of days to get it working. But I've slowly been scripting things and that speeds up the process a lot. Most of my scripts are in Python.
I have a set of opcodes I identify. When I first started doing this, I wanted to keep the list down to the minimum necessary to get showeq working again. For a manual process, that makes sense, the fewer you need to identify, the quicker you finish. However, if you've got a script to do the job, then time becomes much less of a factor. The more opcodes you identify, the easier it is to find other opcodes, using the process of elimination. With the smaller packets, 8-bytes and less, there's so little data to look at. The process of elimination may be all you have to work with.
So this is my current list of what opcodes I look for:
OP_AAExpUpdate
OP_Action
OP_Action2
OP_ClickObject
OP_ClientUpdate
OP_CommonMessage
OP_Consider
OP_Death
OP_DeleteSpawn
OP_ExpandedGuildInfo
OP_Find
OP_FormattedMessage
OP_GroundSpawn
OP_GuildMOTD
OP_GuildMemberList
OP_GuildMemberUpdate
OP_MobUpdate
OP_NewZone
OP_NpcMoveUpdate
OP_PlayerProfile
OP_RemoveSpawn
OP_SendZonePoints
OP_SimpleMessage
OP_SpawnAppearance
OP_SpawnDoor
OP_SpecialMesg
OP_TargetMouse
OP_TimeOfDay
OP_WearChange
OP_ZoneChange
OP_ZoneEntry
On patch day, I started showeq and get it to log zone data. Then I get in the game and do a very specific set of actions that's guaranteed to generate at least one of all of the above opcodes. Then I turn logging off and start digging through the data.
I do this to get a timestamp on the zone.log file:
mv zone.log `date +zone.log.%Y%m%d`
and then I run the following to break up the log file into multiple files, one packet per file:
#!/usr/bin/env python
import sys
import os
def open_outf (work_dir, outfn_n):
outfn = work_dir + '/xx' + ('%06d' % outfn_n)
outfn_n += 1
outf = open (outfn, 'wt')
return (outf, outfn, outfn_n)
if len (sys.argv) != 2:
print 'usage: zone-log-breakout.py <zone-log>'
sys.exit (1)
infn = sys.argv[1]
(_, _, ts) = infn.rpartition ('.')
work_dir = 'work-' + ts
os.mkdir (work_dir)
outfn_n = 0
opcode = ''
(outf, outfn, outfn_n) = open_outf (work_dir, outfn_n)
inf = open (infn, 'rt')
for line in inf:
line = line.rstrip ()
if line == '':
outf.close ()
if opcode != '':
os.rename (outfn, outfn + '.' + opcode)
opcode = ''
(outf, outfn, outfn_n) = open_outf (work_dir, outfn_n)
else:
outf.write (line + '\n')
if line.startswith ('[OPCode: '):
opcode = line.split (' ')[1].rstrip (']')[2:]
outf.close ()
inf.close ()
That gives me a directory named "work-<timestamp>" with all the packets in the log file in separate files named "xx<seqno>.<opcode>".
That's enough to start, we can discuss identifying particular opcodes in other messages.