Struct Comparison for re-alignment?

[Mano]

I believe I have exhausted the search method so now I feel comfortable asking.

Purple mentioned dumping structs for comparison (in the PoR Beta thread) in order to re-align from changes. Is there any way to do such a thing and get an actual formatted name -> value representation?

I'm versed in programming but not familiar with available linux tools / commands.

Thanks.

[KaL]

Wow! Haven't logged into these forums in a year or so... Had to reset my password :)

Anyway, I believe he was referring to dumping structures from EQ itself, on your Windows box.

The easiest way, of course, is to compare an old struct dump with a new struct dump and compare that to the old ShowEQ struct configuration to see how it's done. Of course, I haven't the slightest idea how to do any of that shit.

[purple]

Depending on the struct, I do different things.

For spawnStruct, I usually just mentally lay the struct over the dumps available in the seq zone log. The easiest way to get some mundane things is to zone, capture the spawnStruct from the zone log, then zone back, invis yourself (or turn Show Helm on, or whatever you're testing) then zone again. Then just use the UNIX command diff to find the byte that changed. Once I get most of the struct complete, I use a hacked up version of ksmith's structvis perl script (http://www.showeq.net/forums/showthread.php?t=5041&highlight=structvis) to dump OP_ZoneSpawn to find things like Gender, NPC, Findable, etc. I do have to hack the struct a bit to take out unions and such. But it's just too annoying to keep calculating the offsets to apply to spawnStruct using the huge offsets in ZoneSpawns.

For position structs, there's code in player.cpp and spawnshell.cpp that #if 0'd out which I use to align the positioning structs.

[KaL]

Depending on the struct, I do different things.

For spawnStruct, I usually just mentally lay the struct over the dumps available in the seq zone log. The easiest way to get some mundane things is to zone, capture the spawnStruct from the zone log, then zone back, invis yourself (or turn Show Helm on, or whatever you're testing) then zone again. Then just use the UNIX command diff to find the byte that changed. Once I get most of the struct complete, I use a hacked up version of ksmith's structvis perl script (http://www.showeq.net/forums/showthread.php?t=5041&highlight=structvis) to dump OP_ZoneSpawn to find things like Gender, NPC, Findable, etc. I do have to hack the struct a bit to take out unions and such. But it's just too annoying to keep calculating the offsets to apply to spawnStruct using the huge offsets in ZoneSpawns.

For position structs, there's code in player.cpp and spawnshell.cpp that #if 0'd out which I use to align the positioning structs.

No reverse engineering from the client? Bad-ass.

I've got some time on my hands, maybe I'll play around with that.

[purple]

I'm not smart enough to do that. I wish I were.

[ksmith]

Reverse-engineering from the client is a huge pain (even with offsets from MQ2). I only do it for simple lookup tables or when I can't figure something out by looking at the network traffic.

Edit: Purple, I'd love to get the changes you made to structvis.pl.

[Mano]

Yes, ksmith's perl script is similiar to what I was looking for. It would be real nice if this funtionality was built into the opcode monitor.

This is the first time I have tried to fix seq so it is slow going for me seeing how everything fits together. I'm out of town for a few days, maybe it will be fixed when I get back home.

[Whizzie Wiz]

I've started doing this, and I've identified the Zone Entry code, the CharProfile code and a few others. I'm kicking myself that I don't have a decent zone.log from before the change for comparison....

What are the MINIMUM changes needed to get basic skittles? As in Zone Spawns and 'GPS' functionality?

0999 = CharProfileStruct

spawnStruct is now 773 bytes, and the ZoneSpawn code is 1773 (773 bytes x upto 100), ZoneEntry code is 1220 (773 bytes).

I can also see a lot of 7b59 packets, 22 bytes each, but I cant tell what they are...

[purple]

Minimum functionality to get skittles/spawnlist is probably (off the top of my head):
- Correct opcodes for:
-- OP_PlayerProfile
-- OP_ZoneEntry
-- OP_ZoneSpawns
-- OP_ZoneChange
-- OP_NewZone
-- OP_SpawnDoor
-- OP_NewSpawn
-- OP_DeleteSpawn
-- OP_ClientUpdate
-- OP_Death
- Correct structs for:
-- charProfileStruct (at least zone id, x, y, z, name, level)
-- spawnStruct (at least spawnId, level, x, y, z, deltaX, deltaY, deltaZ, heading)
-- playerSpawnPosStruct (all of it)
-- playerSelfPosStruct (all of it)

7b59 is OP_ClientUpdate. That is the server telling you were a specific spawn id is located. ZoneSpawns is just a lot of spawnStructs. Its size changes depending on how many are in the zone but is always mod sizeof(spawnStruct). ZoneEntry from the server is just the spawnStruct for you, but not everything is filled in.

[dogmeat]

purple, thanks a lot for the information you're providing on how to go about getting things up and running again after a patch. is it safe to assume there's an official purple release on the way eventually?


I'm kicking myself that I don't have a decent zone.log from before the change for comparison....


i hit this exact problem every time i consider trying to fix things. would it make sense to have a current zone log dump in the CVS tree? or, really, after a patch breaks things, the most useful thing someone could post is "here is a zone.log from pre-patch".

thanks again

[purple]

If you want zone logs, why not make them yourself? After I do a release, I always run with zone logs turned on for a day, then save it off, so over time I build up a nice big reference to use. I've thought it would be really nice to have a database of "This is what OP_BlahBlahBlah should look like" and be able to do lookups based on size, but I've gotten by on using grep and my saved zone logs so far.

If every time you want to fix things, you realize you want zone logs, why not turn on zone logs now so next time you have them? That sounds easy enough!

And yes, as soon as I have a working seq I'll release it as usual.

[Cryonic]

Another good reason not to post logs is that the information will contain someone's character information in it.

[Mano]

OP_SpawnDoor = 0db6
OP_PlayerProfile = 0999
OP_NewZone = 2eb6
OP_GuildMemberList = 70fe
OP_GroundSpawn = 7668
OP_MobUpdate = 4f6f???
OP_Animation = 44b1???
OP_ZoneEntry = 1220
OP_ZoneSpawns = 1773
OP_ClientUpdate = 7b59

[Mano]

struct zoneChangeStruct
{
/*0000*/ char name[64]
/*0064*/ uint16_t zoneId;
/*0066*/ uint16_t zoneInstance;
/*0068*/ uint8_t unknown0068[8];

}

Basically this reverted to pre 06/29/2005

struct newZoneStruct
{
...
}

Appended uint8_t unknown0796[4]; to the end to increase size from 796 to 800.

[Whizzie Wiz]

spawnStruct has been totally re-done... trying to figure out the basics at the moment -- current struct bears little resemblance to the older one. If I can locate the spawnId, that should get the rest moving....

[purple]

Yes, spawnStruct looks like this now:



struct spawnStruct
{
/*0000*/ char title[32]; // Title
/*0032*/ uint8_t max_hp; // (name prolly wrong)takes on the value 100 for players, 100 or 110 for NPCs and 120 for PC corpses...
/*0033*/ uint8_t unknown0033;
/*0034*/ float runspeed; // Speed when running
/*0038*/ float walkspeed; // Speed when walking
/*0042*/ uint8_t is_npc; // 0=no, 1=yes
/*0043*/ uint8_t unknown0043[31];
/*0074*/ uint8_t invis; // Invis (0=not, 1=invis)
/*0075*/ uint8_t unknown0075[41];
/*0116*/ int8_t aa_title; // 0=none, 1=general, 2=archtype, 3=class
/*0117*/ uint8_t lfg; // 0=off, 1=lfg on
/*0118*/ uint8_t unknown0118[9];
/*0127*/ char name[64]; // Player's Name
/*0191*/ uint8_t unknown0191[5];
/*0196*/ uint8_t afk; // 0=no, 1=afk
/*0197*/ uint32_t spawnId; // Spawn Id
/*0201*/ uint8_t unknown0201;
/*0202*/ uint32_t guildID; // Current guild
/*0206*/ uint8_t NPC; // 0=player,1=npc,2=pc corpse,3=npc corpse,a
/*0207*/ int8_t guildrank; // 0=normal, 1=officer, 2=leader
/*0208*/ uint8_t unknown0208[4];
/*0212*/ uint8_t beard; // Beard style (not totally, sure but maybe!)
/*0213*/ uint8_t face; // Face id for players
/*0214*/ uint8_t unknown0214;
/*0215*/ char lastName[32]; // Player's Lastname
/*0247*/ uint8_t is_pet; // 0=no, 1=yes
/*0248*/ union
{
struct
{
/*0248*/ uint32_t equip_helmet; // Equipment: Helmet Visual
/*0252*/ uint32_t equip_chest; // Equipment: Chest Visual
/*0256*/ uint32_t equip_arms; // Equipment: Arms Visual
/*0260*/ uint32_t equip_bracers; // Equipment: Bracers Visual
/*0264*/ uint32_t equip_hands; // Equipment: Hands Visual
/*0268*/ uint32_t equip_legs; // Equipment: Legs Visual
/*0272*/ uint32_t equip_feet; // Equipment: Feet Visual
/*0276*/ uint32_t equip_primary; // Equipment: Primary Visual
/*0280*/ uint32_t equip_secondary; // Equipment: Secondary Visual
} equip;
/*0248*/ uint32_t equipment[9]; // Array elements correspond to struct equipment above
};
/*0284*/ uint8_t beardcolor; // Beard color
/*0285*/ uint8_t unknown0285[64];
/*0349*/ signed deltaX:13; // change in x
signed x:19; // x coord
/*0353*/ signed deltaHeading:10;// change in heading
signed z:19; // z coord
signed padding0054:3; // ***Placeholder
/*0357*/ signed deltaY:13; // change in y
signed padding0070:19; // ***Placeholder
/*0361*/ signed animation:10; // animation
signed y:19; // y coord
signed padding0058:3; // ***Placeholder
/*0365*/ unsigned heading:12; // heading
signed deltaZ:13; // change in z
signed padding0066:7; // ***Placeholder
/*0369*/ uint8_t unknown0369[129];
union
{
/*0498*/ int8_t equip_chest2; // Second place in packet for chest texture (usually 0xFF in live packets)
// Not sure why there are 2 of them, but it effects chest texture!
/*0498*/ int8_t mount_color; // drogmor: 0=white, 1=black, 2=green, 3=red
// horse: 0=brown, 1=white, 2=black, 3=tan
};
/*0499*/ uint8_t gender; // Gender (0=male, 1=female)
/*0500*/ uint8_t showhelm; // 0=no, 1=yes
/*0501*/ uint8_t helm; // Helm texture
/*0502*/ uint8_t unknown0502[135];
/*0637*/ uint8_t eyecolor1; // Player's left eye color
/*0638*/ float size; // Model size
/*0642*/ uint8_t unknown0642[4];
/*0646*/ uint8_t level; // Spawn Level
/*0647*/ uint8_t set_to_0xFF[8]; // ***Placeholder (all ff)
/*0655*/ uint8_t unknown0655[5];
/*0660*/ uint8_t hairstyle; // Hair style
/*0661*/ uint32_t race; // Spawn race
/*0665*/ int16_t deity; // Player's Deity
/*0667*/ uint8_t unknown0667[3];
/*0670*/ uint8_t findable; // 0=can't be found, 1=can be found
/*0671*/ uint8_t anon; // 0=normal, 1=anon, 2=roleplay
/*0672*/ uint8_t unknown0672[4];
/*0676*/ char suffix[32]; // Player's suffix (of Veeshan, etc.)
/*0708*/ uint8_t curHp; // Current hp

//wrong but compile
/*0709*/ uint8_t light; // Spawn's lightsource
/*0710*/ uint8_t gm; // 0=no, 1=gm
//end wrong
//

/*0711*/ uint8_t unknown0711[2];
/*0713*/ uint32_t petOwnerId; // If this is a pet, the spawn id of owner
/*0717*/ uint8_t bodytype; // Bodytype
/*0718*/ uint8_t unknown0718[8];
/*0726*/ uint8_t haircolor; // Hair color
/*0727*/ uint8_t eyecolor2; // Left eye color
/*0728*/ uint8_t unknown0728[4];
/*0732*/ uint8_t class_; // Player's class
/*0733*/ uint8_t unknown0733[4];
/*0737*/ union
{
struct
{
/*0737*/ Color_Struct color_helmet; // Color of helmet item
/*0741*/ Color_Struct color_chest; // Color of chest item
/*0745*/ Color_Struct color_arms; // Color of arms item
/*0749*/ Color_Struct color_bracers; // Color of bracers item
/*0753*/ Color_Struct color_hands; // Color of hands item
/*0757*/ Color_Struct color_legs; // Color of legs item
/*0761*/ Color_Struct color_feet; // Color of feet item
/*0765*/ Color_Struct color_primary; // Color of primary item
/*0769*/ Color_Struct color_secondary; // Color of secondary item
} equipment_colors;
/*0737*/ Color_Struct colors[9]; // Array elements correspond to struct equipment_colors above
};
};
/*0773*/


Well except the damn gm flag. If someone wants to find that it would be nice...

[BlueAdept]

After there is a release, Ill fix the light source. Im good at finding that. Bout the only thing Im good at fixing.

[Whizzie Wiz]

Heh.... I feel like such an amateur, I spent all day figuring out just a handful of those new variables. :)

As for the GM flag.... I guess someone could go attend one of those PVP events they're always having in the Arena.... :D

[Whizzie Wiz]

Edit: Purple, I'd love to get the changes you made to structvis.pl.

I found that it was getting confused by some of the comments in the text especially of unions -- if the comment said something like "corresponds to struct above" it basically closed the parsing of the struct there. I tweaked it by forcing it to only accept as a fresh structure definition only those lines that start with "struct whatever" (put an anchor at the start of the regex). Still gets confused in places tho, so I've had to hack chunks out of the everquest.h file to be able to get it to give reasonable results.... and even then it doesnt seem to match what the compiled code thinks the struct is.... :(

[Whizzie Wiz]

Thanks to uRit1u2CBBA='s post of a few days ago, and Purple's struct update above, now I see skittles again :) unfortunately the playerUpdate codes seem borked, so the map quickly blows up off the scale. Even before I fix that, it's already helping me find the damn tradeskill vendors for all my trophy bits... :D

[purple]

Even once you get the position stuff fixed you're not to the hard part 8) I had that done on Thursday!

[dogmeat]

OP_Death 640b
OP_NewSpawn 44A3
OP_DeleteSpawn 33B6




struct playerSpawnPosStruct
{
/*0000*/ uint16_t spawnId; // spawn id of the thing moving
/*000 **/ signed deltaY:13; // change in y
/* 2**/ signed x:19; // x coord
/*0000*/ signed deltaHeading:10; // change in heading
/* */ signed z:19; // y coord
signed padding0002:3; // ***Placeholder
/*0010**/ signed deltaZ:13; // change in x
signed deltaX:13; // change in z
signed padding0006:6; // ***Placeholder
/*0014**/ signed animation:10; // animation
/*0014 **/ signed y:19; // z coord
signed padding0014:3; // ***Placeholder
/*0018**/ unsigned heading:12; // heading
unsigned padding0018:20; // ***Placeholder
/*0022*/
};


struct playerSelfPosStruct
{
/*0000*/ uint16_t spawnId; // Player's spawn id
/*0002*/ uint8_t unknown0002[2]; // ***Placeholder (update time counter?)
/*04*/ signed deltaHeading:10; // change in heading
signed padding0004:6; // ***Placeholder
/*0006*/ uint8_t unknown0006[2]; // ***Placeholder
/*0008*/ float deltaZ; // Change in z
/*0012**/ float y; // y coord
/*0016*/ signed animation:10; // animation
/*0018*/ unsigned heading:12; // Directional heading
unsigned padding0020:10; // ***Placeholder (mostly 1)
/*0020**/ float x; // x coord
/*0024**/ float deltaY; // Change in y
/*0028**/ float deltaX; // Change in x
/*0032**/ float z; // z coord
/*0036*/
};

[dogmeat]

Auras are screwing things up. Looks like a OP_ZoneEntry(server) is sent for them, which makes the SEQ client think it's somewhere in the zoning process. I turned them off by looking for "Aura_" or "_Aura" in the spawnstruct->name in ZoneMgr::zoneEntryServer(), and just bailing if I find that. It's hacky, but it works for now. A cleaner solution would involved indentifying a bit/byte in the spawnstruct that is set for Auras.

[Mano]

Prior to the update by purple to fix skittles occasionally when zoning the player name would be the name of an aura (in the spawn list and on the map) but have all the same attributes of an actual player (the amount of hp, location, etc)