Erratic Performance - 4.3.10

[wiz60]

I have looked around to see if anyone has posted a similar topic. It seem slike nothing has been posted yet.

I have version 4.3.10 working on my system. It seems to decode the zone fine when I first zone in. However - I lose track of some PCs when they run around - I can't pin it on distance - because they do not come and go on the display.

Also it seems that I do not see new mobs as they spawn - or that new spawns turn up as "unknown" when they do.

I minimized the main seq screen - and noticed that I was getting errors related to BAD CRC - and packets being discarded. This was not happening continuously - but on occassion. The screen was by no means flooded.

I am running on a P2-400 class machine. My primary suspicion is the processor can't keep up with the activity. I have ordered a new MB and CPU to fix it - but wanted to pursue this forum in parallel.

Has anyone else seen the problem?

Are there any work arounds for it?

[LordCrush]

serach -> Unknown Spawn

http://seq.sourceforge.net/forums/showthread.php?s=&threadid=3929

[wiz60]

Thanks for the reference.

I checked the Development and General Discussion forums.

Never thought to look at Help Desk. I honestly believe this is a development issue - not a "how do I do it" issue. While I am not above questioning my own abilities - I have been using SEQ since it's HackersQuest days. I really did not expect any user to solve this problem. I am relieved that others are seeing it also.

LOL - had I posted this first I could have saved the money I am spending on a MB upgrade - oh well - I will have lightning fast Linux - with Unknown Spawns.

[LordCrush]

Hehe,

i am runnig SEQ with 3 or 4 Sessions on an Athlon 500 / 512 MB worke fine just before the last patch.

But i had a similar looking Problem some time ago and then it was a defect Hub ...


btw ... i find its good placed here too, after some more people have that problem

-- LC

[S_B_R]

I regularly run 2 SEQ seesions on a 400MHz Celeron with 256meg of ram. Before that I regularly ran 2 SEQ sessions on a 233MHz Cyrix with 64meg of ram.

What I'm saying is, CPU performance isn't a problem on any machine built in the last 6 years or so... ;)

[CiscoKid]

Originally posted by S_B_R
I regularly run 2 SEQ seesions on a 400MHz Celeron with 256meg of ram. Before that I regularly ran 2 SEQ sessions on a 233MHz Cyrix with 64meg of ram.

What I'm saying is, CPU performance isn't a problem on any machine built in the last 6 years or so... ;)

Heh, you mean I can run 2 sessions to track both toons?? LOL didn't know that! :p

[S_B_R]

Yeah, if you're 2 boxing using 2 seperate machines it works great. Just point one instance of SEQ at each IP address. If you are 2 boxing on 1 PC it works but it's a little less clean.

[monklett]

I am running .10 on a pentium-233, and I have updates cranked down to 5 fps, but it works fine.

Based on your symptoms, I would suspect hardware issues. Start by looking at your system logs; you may have a dead/dying network interface card.

[S_B_R]

nah the unknown spawns thing is a known problem, since the last EQ patch.

[monklett]

Packet CRC errors sounds like a hardware issue. <shrug> Wasn't aware that it was a known issue; it seems to work fine for me.

I am running .10 with the /con patch and the bazaartracker patch.

[Hendrix_Morton]

Originally posted by S_B_R
Yeah, if you're 2 boxing using 2 seperate machines it works great. Just point one instance of SEQ at each IP address.

OMG...I never thought of that...Sometimes the simplest things can be so hard to see...

[LordCrush]

I would like to bring this back from the discussion about 2-boxing to the real problem :D

@monklett: Yes it *could* be somthing with the hardware. I had a similar problem in February and it resolved as a bad hub. But the main difference to that was, that i was the only one who had that error.
And i am getting the same error too if i run SEQ in a VMWare session on my notebook that i use for packet analysis while work... the NIC in the notebook works and the VMInstallation of SEQ works too (I have checked that with a friend who does not have the problem).
Anyway i will change my hub today evening just to test that. Need to oragnize a "new" and "real" Hub first :p

On the other hand ... what can we do to track that ugly bug down ?

I am not familiar with the packet decoding part of SEQ ... perhaps someone from the DEV-Team can give a hint where to place some debug code or such.

Let me again sumon up the symptomes from the other thread


Complied 4.3.9 yesterday just fine, on initial zone-in everything works and i get colored spawns.
But every spawn after this is shown as unknown.

Running SUSE 7.3, Gcc 3.03, Qt 3.0.5


Same here. The initial dump of mobs is fine, but the new spawns are not quit correct. I've also noticed the code for removing ground spawns after being picked up are not working.


I get only unknown spawns after initial Zonein ... if i turn of "Create unknown spawns" i see no new spawns


Hmmmm, I was in PoV lastnight for most of the time, and cleared the quite a few mobs, all of which respawned an known spawns. Not really much of a test I know, where I did see unknowns was in PoK lot's a of PC traffic through there.


Still getting only unknown spawns after intital zonein .. with the new Version 4.3.10 too

The output on the console looks everytime like this, but it is not the same opcode every time:




--------------------------------------------------------------------------------
Loaded map: '/usr/local/share/showeq/Poknowledge.map'
No Zone Specific filter file '/usr/local/share/showeq/filters_poknowledge.conf'.
Loading default '/usr/local/share/showeq/filters.conf'.
Your player's id is 2344
WARNING: MakeDropCode (00f3) (dataLen:2 != sizeof(makeDropStruct):94)!
dispatchSplitData(): WARNING OpCode 0xc552 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0008 before completion of 0x0007
Lost sync, relog or zone to reset
uncompress failed on 0xb2c5: data error
no further attempts will be made until zone.Player::backfill():
Loaded map: '/usr/local/share/showeq/Potranquility.map'
No Zone Specific filter file '/usr/local/share/showeq/filters_potranquility.conf'.
Loading default '/usr/local/share/showeq/filters.conf'.
Lost sync, relog or zone to reset
uncompress failed on 0xb2c5: data error
no further attempts will be made until zone.Your player's id is 1514




I had it happen to me last night with 4.3.9 upon zoning into Dreadlands. Funny thing is, I had just came from there and was there approx. 5-6 hours with spawns respawning as known the whole time, and then bugged when I came back from banking in PoK. So it's not a for sure thing that it will always bug like that, at least from my experience last night.


Same here, upon zone in, I get all the skittles but every new spawn is unknown.

Tested Zones: Nadox, Grimling Forrest, Acrylia Caverns

Tarball: 4.3.9 and 4.3.10


Have tried it on new SUSE 8.2/8.3 but works not better ... so no outdated library :/


Same issue on Redhat 9....everything is fine upon zone in, but everything (Players and Mobs) all show up as unknown after that...

[BadgerOne]

It might be a Hardware Issue... or just coincidence.

But yesterday my USR Broadband Router stopped working and went into transparent mode (routing the WAN port direct to the built-in switch) and guess what: I got no more unknown spawns and corpses turned out to be crosses again.

[LordCrush]

Hmm ... trying today evening with ISDN-Connection ... I use a Windows 2000 Server as Router

--- YES --- ;)

[devnul]

It's new. It's real.

Until this patch I never had unknown spawns either. My hardware has not changed. My software has not changed. Recently I have only used Linux for SEQ and nothing has changed on this box.

It's not only players either. Summonned a pet and it was unknown.

devnul

[EnvyEyes]

I seem to remember a similar problem when the additional bank slots were added. Didn't some people have problems with decoding new spawns, while others would work fine?

I may be thinking of a different issue here, but I'm not at home and don't have time to track down my theory. I, too, am having the same issue as described above (both PCs and NPCs are showing as unknowns). I also haven't had a chance to work on it due to RL issues. I'll be happy to try anything worthwhile that is thrown my way.

[Hendrix_Morton]

Hmmm...Well, they did add the shared-plat bank slot last patch...

[BlueAdept]

I didnt notice any problems...until today in the nexus. I tried to con my friend and found she was an unknown. It doesnt seem to happen all the time...just some times.

[Zaphod]

Blah.... I have some suspicions about these unknowns....
Looking at LordCrush's post in the other thread I see two indications of bad juju magic occuring. First, where it complains abut recieving a new fragment before completion of another:

dispatchSplitData(): WARNING OpCode 0xc552 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0008 before completion of 0x0007

This is generated in packet.cpp line 1568 and indicates that showeq is seeing a new fragment packet sequence begin before the packet containing the last fragment of the current fragment packet sequence has been recieved. This could either be caused by:
The drop of a packet
A large number (greater then ArqSeqGiveUp) number of outstanding packets causing packets to be skipped (to keep ShowEQ from waiting infinitely on a dropped packet).
Some reason SOE started sending down multiple packet streams.


Some requests for those seeing the problem:[list=1]
What is your current ArqSeqGiveUp setting value (it can be found under "Network->Advanced->Arq Seq Give Up->")?
Are you noticing lag spikes in EQ, lots of red in the ole lag-o-meter or the like?
Does increasing the ArqSeqGiveUp setting by 64 or 128 alleviate the problem?
Are any of you brave/insane enough to go into packet.cpp and uncomment PACKET_PROCESS_DIAG, PACKET_CACHE_DIAG, and/or PACKET_PROCESS_FRAG_DIAG, recompile, and then sift through the output of a session?
[/list=1]


The other message that concerns me is:


Lost sync, relog or zone to reset
uncompress failed on 0xb2c5: data error


These errors comes from libEQ.cpp (lines 174 and 177 respectively). The "uncompress failed on:" message occurs whenever a compressed packet fails to uncompress. The "Lost sync, relog or zone to reset" indicates that the failure to uncompress occurred on a packet that was supposedly encrypted. If somethings screwy here it may be related to some subtle change in how compressed/encrypted data is being sent down (at least in some instances). I'm not saying that SOE has changed anything in this regard, but I'm also not yet familiar with the new encryption/decoding code. This could also possibly be caused by dropped packets and how EQPacket::dispatchSplitData doesn't really take them into account.

Enjoy,
Zaphod (dohpaZ)

[fester]

Zaphod, "Lost sync, relog or zone to reset" is a result of "received new fragment seq 0x0008 before completion of 0x0007" because you only know the new encryption XOR value if you properly processed the previous packet.

If a packet is lost, there is NO clean way to resync.

The easiest way to resync is to get a channel message opcode that is either a single packet of a combined packet of ONLY channel message packets. Due to the nature of how channel messages structures are organized you can verify or resync to the current XOR value because a section (that was XOR'd) is always 4 nulls (0x0) and thus will be the "plaintext" XOR key.

[bonkersbobcat]

As another data point, I am not seeing this behavior at all. Running RedHat 9, with current CVS, works like a champ. I haven't had a lost sync in months.

I do have a really stable network connection. Troubleshooting focus on solving this should probably be looking at how SEQ reacts to flaky networks, and lost packets. (Or, of course, improving the network...)

[LordCrush]

Thank you all for reply

My "Arg Seq Give up" was set to 256, - changed it to 512 ... but no change

To my network setup:

I use a DSL leased line with a windows 2000 SP2 Server as router to the internet. Atm i have no firewall or packetfilter enabled *couch* :p.

The internal setup is as follows:

The Router and some other Servers are connected to a 10/100 Switch and from there i have an uplink to a 100 MBit real hub where the Linux pc and 3 EQ-Clients are attached.

I run 2 or 3 Sessions of Seq on this "SEQ-Server" and use VNC to view Seq from my Notebook.

I have tried with a SEQ running in on my Notebook in a VM-session. In this case i do not need VNC. But it was the same.

I have no problems with packet loss or drop packets with EQ both indicators sho 0.0% normaly.

Hmm .. i will try a new hub today ...

But i think i will not do further debuging before the LoDN Patch on Tuesdays is through :p

[BlueAdept]

Linux box is directly connected to the Inet.

I have a 10/100 hub/switch with the linux box, and 2 other computers hooked to it.

I didnt experience any noticable lag spike. It also happened to me in another zone. I had some mobs on the inital zone in, but it didnt even get to the drops/ground spawns/forges/etc before it stopped decoding.

[fester]

I had this problem when I had everything on a 100mb hub.

When I troubleshot the problem, it was that 100mb was not enough to support peak usage (if I was doing a file transfer over the network while EQing the 100mb would reach peak.)

The fix is to configure the network where my sniffer/showeq box is sitting in between me and the Internet (where I only have 1.5mb) so it will never reach peak (at least not on a 100mb connection.)

Also it would only happen rarely (say once every couple hours.)

[S_B_R]

I bumped my ArqSeqGiveUp up to 384 from 256 and I didn't have any unknows for roughly 2 hours of play. One other thing I need to check on is, normally I 2box (with 2 seperate PC's) and I was only playing 1 character durring this 2 hour test session.

Normally no lag spikes, packet loss is almost always 0.0%. I run SEQ on my Router/Firewall box.

[minigirl]

Discovered this problem also a few hours ago.....

i run MySeq side by side with Showeq. I use the soundalert from MySeq to indicate waht i need...

well during some days vacation i have had my main eq box in the livingroom...that one is on wifi to the 10mb hub and that hub is uplinked to my wan router.

from the hub i also run my linux box and whathave we...


when i move my gear down to the basement again i switched back to PDS wire for the eq box ...nothing was changed on the linux box.

i did not get skitties on update..only unknowns.


i activated wifi again on eqbox, booted main wan router and restarted eqbox....all is fin..now however i have packet lag on the eqbox ( due to wifi - but only about 5 pct ).


whatever the trouble is...it must be hardware issues....

i did solve my own problem..but sacrified the faster wired lan connection over the wifi to get it all working.


Mini

[minigirl]

i use VNC to access linux box and access the VNC server using two separate Laptops so i have a console in two rooms.


It do not appear to be a VNC related problem.

[BadgerOne]

I replaced the Network card on the SEQ Box and everything works again. It seems that the the combination of my defective router and the the old nic caused the problem. So I would say the problem is caused by Hardware. I am not at home atm. to chec the ArqSeqValue, but I will once I get home.

[LordCrush]

Ok ... what i see from this thread ... it seams to be somewhat related to the Hardware / Networksetup.

But there must be something to with the software, i don´t belive that we all have hardware failures after the 08/26/03 :p

I will test soon with another Hub and i can change my NIC too. But perhaps i will not be able to do this today and then the LoDN patch inc ...

[Edael]

Hi guys,

I think it has something to be with zoning, It detects that EQ starts to zone but doesnt detects well when the EQ stops zoning and normal play begins.

I removed the if (stillZoning) return; (dont renember exact var, as im at work :( ) from the newSpawn signal code, and not a single unknown since then.

It still "hangs" while zoning and stops processing new zones, have to restart SEQ from time to time.

Ill try the .11 version as soon as i get home

Continue the great work :)

[S_B_R]

In my case it's impossible for it to be a hardware problem. I'm running SEQ on my router/firewall box, all packets pass through it. Therefore if it were a problem with the hardware I would be getting disconnected from EQ do to packet loss or corruption.

I still need to test more but, lastnight I was 2 boxing and I did get unknows, even with ArqSeqGiveUp set to 384...

[Zaphod]

Originally posted by Edael
Hi guys,

I think it has something to be with zoning, It detects that EQ starts to zone but doesnt detects well when the EQ stops zoning and normal play begins.

I removed the if (stillZoning) return; (dont renember exact var, as im at work :( ) from the newSpawn signal code, and not a single unknown since then.

It still "hangs" while zoning and stops processing new zones, have to restart SEQ from time to time.

Ill try the .11 version as soon as i get home

Continue the great work :)

The zoning state machine seems to still be working as far as I can tell. Removing the isZoning() calls from SpawnShell can have some ugly side effects. The reason the whole zoning state machine and isZoning() system exists is because between the sending of the ZoneChangeCode and the ZoneEntryCode ShowEQ may still receive a number of packets for the previous zone (including, but not limited to, PlayerPosCode, MakeDropCode, RemDropCode, NewSpawnCode, and DeleteSpawnCode). If ShowEQ processes these results it can lead to unpredictable behavior. The only way that the isZoning() will continue to be true is if your client didn't see the ZoneChangeCode and the NewZoneCode.

I'm not saying there isn't a problem, I just don't think that is it.

Enjoy,
Zaphod (dohpaZ)

[Edael]

SEQ runs at my firewall, so i cant lose any packet, like s_b_r (2 ethernets config)

What happens to me is that i miss the ZoneChange or the ZoneEntry, sometimes it locks in "--- Busy zoning--- " state othe time i zone and works ok but i get the unknowns because the "isZoning()" call stills returns true.

Dont know why, im just trying to understand the whole thing, and its damn hard (i think i just need to do some paperwork and draw a schema of the protocol, and it will be more clear)...

Thanks for the explanation.

[Zaphod]

Originally posted by Edael
SEQ runs at my firewall, so i cant lose any packet, like s_b_r (2 ethernets config)

What happens to me is that i miss the ZoneChange or the ZoneEntry, sometimes it locks in "--- Busy zoning--- " state othe time i zone and works ok but i get the unknowns because the "isZoning()" call stills returns true.

Dont know why, im just trying to understand the whole thing, and its damn hard (i think i just need to do some paperwork and draw a schema of the protocol, and it will be more clear)...

Thanks for the explanation.

Some comments/questions:
Just to be clear, just because your machine sees a packet doesn't mean pcap and therefore ShowEQ will. If for some reason the thread that is receiving the pcap information doesn't returned quickly enough packets will be missed since the OS will only keep so many in queue. This shouldn't happen (and is why our packet capture is in its own thread), but I have seen it happen before (especially when debugging the pcap callback).
Are their any particular zones people are seeing this happen in?
As to working through the protocol, feel free to, we could always use another set of eyes. The biggest pain with this protocol is that it is a moving target. The more eyes the better I always say... ;)

I just noticed that we could be truncating the incoming packet. Could someone who is experiencing this problem and doesn't mind editing code do a test for me? If so here it is, change both instances of the number 1500 in packet.cpp to BUFSIZ. Such that packet.cpp line 769 appears as:

unsigned char buffer[BUFSIZ];


and that packet.cpp line 3506 appears as:

m_pcache_pcap = pcap_open_live((char *) device, BUFSIZ, true, 0, ebuf);


Then re-compile, run, and tell us what happens.

Do we have any specific messages occuring before/after we lose skittles? I want to see if we can come to a consensus. Such as "dispatchSplitData(): WARNING OpCode 0xc552 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0008 before completion of 0x0007", or "Lost sync, relog or zone to reset", and/or "uncompress failed on 0xb2c5: data error"?



Enjoy,
Zaphod (dohpaZ)

[S_B_R]

Back when this whole thing came up I was in PoV and I didn't notice any NPC unknows after having cleared the west wall camp a couple times. The times I've noticed it the most is in PoJ. But again I don't have a very wide sample to really be useful.

I've changed my packet.cpp and I'm compiling now. I'm at work now so I won't be able to do any testing till later tonight.

[S_B_R]

2 hours in PoD and no unknowns, With 4.3.11 with the packet.cpp changes from above.

[S_B_R]

ok, a little over an hour in PoJ and I got unknowns here's what was in the terminal
Lost sync, relog or zone to reset
uncompress failed on 0xb125: data error
no further attempts will be made until zone.EQPacket: SEQClosing detected, awaiting next zone session, pcap filter: EQ Client 192.168.0.5
Clearing Cache[zone-client]: Count: 1
Resetting sequence cache[zone-client]

[Zaphod]

Originally posted by S_B_R
ok, a little over an hour in PoJ and I got unknowns here's what was in the terminal
Lost sync, relog or zone to reset
uncompress failed on 0xb125: data error
no further attempts will be made until zone.EQPacket: SEQClosing detected, awaiting next zone session, pcap filter: EQ Client 192.168.0.5
Clearing Cache[zone-client]: Count: 1
Resetting sequence cache[zone-client]

While I was examining your output and looking at decodeOpCode() I found several pre-existing bugs in libEQ.cpp that are apparently now being triggered by changes in the EQ datastream:
[list=1]
With the error messages that you indicate, the real opcode being processed was 0xf125 (not 0xb125). This was a bug where the opCode was being modified too early, but really only effects the accuracy of the error message displays.
Even on an uncompress failure on an unencrypted packet (FLAG_CRYPTO not set) would still set valid_key to false, even though the key wasn't effected.
The packet causing the error shown above is a FLAG_COMP | FLAG_COMBINED | FLAG_IMPLICIT | FLAG_CRYPTO version of opcode 0x0125 (SpawnUpdateCode) which from my experience is only sent from the client to the server. If in fact encrypted packets are being sent from the client to the server (and they used to occasionally in the bad old libEQ.a days) it would fail since it would be using a server decode key to decrypt a client packet. This would also have the side effect of screwing up the server side decode_key since it will have been modified by the contents of the client packet.
[/list=1]

I've coded up a fix for problems 1 and 2, but I'm still deciding how to approach 3.

What would be great is if someone could create a zone log file with the new 4.3.11 raw packet logging enabled and then tried to generate the error. They could then search the generated zone.log file for the offending packet and see which direction it was coming from and inform us here.

Thanks and Enjoy,
Zaphod (dohpaZ)

[minigirl]

And not a single unknown with 4.3.11 :=)



also 8 hours in Bazaar...not a single unknown.

/bow

[Catt]

I have not gotten the latest yet, however, I did have some troubles as described above, lost sync, unknowns. I had been running flawlessly for days, then the above started.

I found it was related to network packet loss, I had a router that was a bit wonky, which after some adjustment and rebooting, everything is/was back to normal. I played about 14 hours yesterday with NO troubles.

I will be updating shortly to .11, and hope as I'm sure most do that things still work post EQ patch (in progress).

[IamGman]

~ $ /usr/local/bin/showeq
ShowEQ 4.3.12, released under the GPL.
SINS 0.5, released under the GPL.
All ShowEQ source code is Copyright (C) 2000-2003 by the respective ShowEQ Developers
Binary distribution without source code and resale are explictily NOT authorized by ANY party.
If you have paid for this software in any way, shape, or form, the person selling the
software is doing so in violation of the express wishes and intents of the authors of this product.

Please see http://seq.sourceforge.net for further information

Using config file '/usr/local/share/showeq/showeq.xml'
Loaded preferences file: /usr/local/share/showeq/seqdef.xml!
Loaded preferences file: /usr/local/share/showeq/showeq.xml!
Listening for first client seen.
Initializing Packet Capture Thread:
Filtering packets on device eth0, searching for EQ client...
Running without libEQ.cpp linked in.
Loading filters from '/usr/local/share/showeq/filters.conf'
Loading Filter File: /usr/local/share/showeq/filters_unknown.conf
GuildMgr: guildsfile loaded
Categories Reloaded
Error opening map file '/usr/local/share/showeq/unknown.map'!
Opcode Logging Mask: 0 0 0
Client Detected: 192.168.254.1
Player::backfill():
Loaded map: '/usr/local/share/showeq/Poknowledge.map'
No Zone Specific filter file '/usr/local/share/showeq/filters_poknowledge.conf'.
Loading default '/usr/local/share/showeq/filters.conf'.
TIME: 06:16 07/27/3172
EQ EPOCH OCCURRED AT 812747766 SECONDS POST UNIX EPOCH
Your player's id is 449
Lost sync, relog or zone to reset
uncompress failed on 0x52c5: data error
no further attempts will be made until zone for direction 2.
dispatchSplitData(): WARNING OpCode 0xc5f2 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0011 before completion of 0x0010
dispatchSplitData(): WARNING OpCode 0xc5f2 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0013 before completion of 0x0011
dispatchSplitData(): WARNING OpCode 0xc552 will not be processed due to loss
dispatchSplitData(): recieved Out-Of-Order fragment seq 0x0001 (0x0015) expected 0x0001
dispatchSplitData(): WARNING OpCode 0xc552 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0016 before completion of 0x0014
EQPacket: SEQClosing detected, awaiting next zone session, pcap filter: EQ Client 192.168.254.1
~ $

This is what I get when I run the 4.3.12. I am particularly interested in the part Running without libEQ.cpp linked in. Is this normal? Thanks for your help in advance.

[LordCrush]

The part with libeq.cpp is ok ...

Same still here with 4.3.12

... will check hardware today

[minigirl]

Changed network card in seq/ux server. Still unknowns.

Changed upstream hub before WAN/DSL box. Still unknowns.

Then I changed to use 3com wifi network interface on eq box again instead of the 3com ordinary lanwire card. The performance on this ( the wifi card ) I do see as the less "good" one. But no unknowns seen now.

hmm..must try change to another ordianary lan card


hmm...

[BadgerOne]

I have still problems, even after replacing faulty Hardware. Yesterday the maps stopped loading upon zoning.

But that was still with 4.3.11

I'll try today with 4.3.12

[LordCrush]

Originally posted by Zaphod

What would be great is if someone could create a zone log file with the new 4.3.11 raw packet logging enabled and then tried to generate the error. They could then search the generated zone.log file for the offending packet and see which direction it was coming from and inform us here.

Thanks and Enjoy,
Zaphod (dohpaZ) [/B]

i will try to get this on the weekend

and again THANK you for your great work :)

/wave

[S_B_R]

RePost from this (http://seq.sourceforge.net/forums/showthread.php?s=&threadid=3998) thread:
Little more info, SEQ 4.3.12, Zone EC
dispatchSplitData(): WARNING OpCode 0xc552 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0010 before completion of 0x000f
dispatchSplitData(): WARNING OpCode 0xc5f2 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x0011 before completion of 0x0010
Lost sync, relog or zone to reset
uncompress failed on 0xf2c5: data error
no further attempts will be made until zone for direction 2.

[LordCrush]

Originally posted by Zaphod

What would be great is if someone could create a zone log file with the new 4.3.11 raw packet logging enabled and then tried to generate the error. They could then search the generated zone.log file for the offending packet and see which direction it was coming from and inform us here.

Thanks and Enjoy,
Zaphod (dohpaZ) [/B]


Originally posted by LordCrush
i will try to get this on the weekend

and again THANK you for your great work :)

/wave

Hmm I dont find that Zone.log file =/ .. i have activated

Network->Log->Raw Data

Save Pref, Shut down SEQ, restart SEQ, Zone within EQ ... no zone.log file

[LordCrush]

Ok ... i have the Zone.log file now, but how do i find the "offending" Packet =p... the Zone.log shows no "error" in any aspect for a n00b like me :D

:confused:

Any help is welcome =)

[Hendrix_Morton]

Over the past week, I have swaped out my NIC card (twice), the HUB (twice) and the DSL Router that I use and still get everything that spawns/zones-in after I zone in showing up as unknowns...

Want to help as much as my limited time allows, so whats the next step to help those that can debug something like this?

>HM<

[Zaphod]

Originally posted by LordCrush
Ok ... i have the Zone.log file now, but how do i find the "offending" Packet =p... the Zone.log shows no "error" in any aspect for a n00b like me :D

:confused:

Any help is welcome =)

Find the packet that has the same OpCode that you saw getting a warning about on the screen, so if you saw S_B_R's "dispatchSplitData(): WARNING OpCode 0xc552 will not be processed due to loss" then you sould search for a packet beginning with
"000 | 52 c5" (000 is the beginning of the packet, and c5 and 52 are reversed due to the endiancess of the network transmission). So, if you were viewing the file with the 'less' pager you would do a search of "^000 \| 52 c5".

Thanks and Enjoy,
Zaphod (dohpaZ)

[LordCrush]

I have found the bad packet ... I hope the dev people can see somthing from it




Decoded - Server->Client: (20) Tue Sep 16 19:17:44 2003
000 | 22 00 54 08 fb 05 00 05
CUT ...

Raw - Server->Client: (579) Tue Sep 16 19:17:44 2003
000 | c5 52 78 5e c5 88 f2 88 96 06 69 6f 1e 28 55 ee | .Rx^......io.(U.

....

576 | 5a 74 2e | Zt.



Tue Sep 16 19:17:44 2003
- Uncompress failed : - Server->Client:
Lost sync Server->Client - Decrypt Key is invalid now
Uncompress failed on 0x52c5: data error
No further attempts will be made until zone for direction 2.



I have cut out the most of the packet, i pm it if needed.

Here is the corresponding Console output:



TIME: 21:39 08/13/3175
EQ EPOCH OCCURRED AT 808814175 SECONDS POST UNIX EPOCH
Your player's id is 2735
SPELL: You begin casting Immolate. Current Target is ein_riesiger_Blattskarabäus01(2260)
selfStartSpellCast - id=78
SpellItem 'Immolate' finished.
Tue Sep 16 19:17:44 2003
- Lost sync, relog or zone to reset, still encrypted
Tue Sep 16 19:17:44 2003
- uncompress failed on 0x52c5: data error
no further attempts will be made until zone for direction 2.
dispatchSplitData(): WARNING OpCode 0xc5f2 will not be processed due to loss
dispatchSplitData(): recieved Out-Of-Order fragment seq 0x0001 (0x000c) expected 0x0001
dispatchSplitData(): WARNING OpCode 0xc5f2 will not be processed due to loss
dispatchSplitData(): recieved new fragment seq 0x000d before completion of 0x000b
Clearing Cache[zone-client]: Count: 1
Resetting sequence cache[zone-client]


I coded some debug-log-code into the libEQ.cpp that it writes to the zone.log and the world.log

I have attached the libEQ.cpp with debug code to this post.

I hope this helps in further debuging.

-- LC

[LordCrush]

Here is the file :p

[minigirl]

look above...exactly the same behaviour i had earlier. I do get a few unknowns still...but much less now since i have gone to another ( inferior ) nic.

I can force this behavior to show again if i wanted....which i do not...:P

[BadgerOne]

I was under the impression earlier, the problem is caused because of Hardware. Well, I tried all my available Hardware Options and I get unknowns about 90 Percent of the times after I zone.

Each and everytime the same Output as Lord Crush has reported.

Are the people that never ever have Unknowns ?

I guess LC is from Europe as I am. Is it possible this could have something to do with it ?

[LordCrush]

Germany - Kael Drakkal

[wiz60]

The problem is NOT related to Europe.

I started the thread. I am based in the US and play on E'ci - hence it is NOT a server problem.

My problems are identical.

As to equipment - I am using all name brand equipment.

3COM NICs - Dlink HUBs - Verified Belkin cables - CAT-5e.

The equipment is unchanged - and the problem seems to have started with one of the more recent patches - approx 4.3.9/10 version number.

I am using RedHat Linux V9.0 native - no other patches - or software versions.

[S_B_R]

I run into this situation alot more when I 2box. Actually I can't think of 1 time this has happened when playing only a single Character. I've even taken to not running a second SEQ session for my second account when I 2box still same results.

[wiz60]

I can 3 box - and see the problem equally when 1 boxing, 2boxing, or 3 boxing.

[fester]

I do not have this problem when I play. I have 4 accounts. My roommate has 5 accounts. We have 9 computers here. We have all 9 firing at once and this does not happen for me on US servers.

It DID happen in the past.

I had this problem when my switch (cisco 2924) was sending all traffic from all ports to my ShowEQ sniffer port (my ShowEQ systems has two ethernet ports, one for it's traffic and one devoted to sniffing.) I fixed it by having it send only Internet traffic to the sniffer port (the one port that goes to the Internet firewall in the other room.)

I have not had this problem since and I have played many hours. When I did have the problem, I noticed the counters on the switch for "dropped packets due to buffer full" were incrementing. Basically 100 mb was not enough pipe to copy all packets and some EQ packets were certainly going to be dropped.

What this means, is that if your ISP, hub, firewall, nics, etc have a chance to drop packets, this problem will increase?

[wiz60]

I have approx 8 nodes on my network. But I am the only user. I suspect there is traffic related to the windows infrastructure that is not obvious - but I think the non-Internet traffic on my network is negligible. In any case - it has been there for the 4 years I have run SEQ - and has not been a problem in the past.

The funny thing in virtually all these cases - the issues relate to processing new spawns - be they NPCs or people zoning into the zone. It seems to be a packet TYPE issue. If the issue was related to network architecture - you would expect it to interfere in other ways - for instance the occassional failure to decode a zone - or losing track of my movement on the screen. These issues NEVER happen.

From time to time - I have seen it lose track of where a known PC is. I see the PC represented on the screen with a large velocity line - but NOT moving. They will freeze in that position - until something changes - for instance they die. Then they will vanish all together.

[S_B_R]

The reason I can't believe it's a Hardware/Network problem is, how is it possible that all of us experiencing this problem had the exact same hardware/network failure presicely when SEQ 4.3.10 was released?

[LordCrush]

/agree S_B_R

i had a similar problem in february i think - but i was the only one who had it - and that was a buggy hub, but now ... i tried to switch hardware, but nothing changed

[Dedpoet]

I have 4 accounts. My roommate has 5 accounts. We have 9 computers here. We have all 9 firing at once and this does not happen for me on US servers.

Fester, you have a problem alright, and it doesn't have to do with TCP/IP packets :-P

[LordCrush]

Is there anything more that i can provide to help debuging this ?

Thanx in advance

-- LC

[BadgerOne]

are people still getting unknowns ? or is replacing BUFSIZ with 1500 the way to go ? Or is more info needed ?

[LordCrush]

Did not help me ... still all unknown after initial Zonein.

I made the following tries yesterday:

1.) I changed my Hub -> Still unknowens
2.) I used a SEQ 4.3.12 that is installed in a VMWare Linux Suse 8.0, that i got from a friend. This SEQ works flawless at my friends home, but i get unknowns. We have the same Notebooks with the same hardware equipment (P4 2,8 GHz, 1GB Ram) so ist not the NB that is too slow and we use the same VMWare version :p

Hmm looks like something hardware- or network-related. I will try with annother NIC for my router and perhaps do a networkscan for retransmits ..

[Hendrix_Morton]

Originally posted by BadgerOne
are people still getting unknowns ? or is replacing BUFSIZ with 1500 the way to go ? Or is more info needed ?

Still getting them here, and the BUFSIZE issue is in the .12 version, so that doesnt help...

I swaped out all the hardware involved, some 3 and 4 times and still get the unknowns...

LordCrush was doing some packet diagnostics to try and detect what might be causing it, but havent heard anything from the developers about this in awhile..

/em wishing there were more than 24 hours in a day...

[S_B_R]

Still getting it here as well...

[Zaphod]

I'm having trouble getting a datastream that will produce the described behavior in 4.3.12. So if someone really wants to help and doesn't mind giving up some level of their annonymity to me they can e-mail me a packet recording that exhibits the problem.

To do this you would use the '--record-filename=<FILENAME>' option when running ShowEQ and go through a session and attempt to reproduce the problem. If you are able to reproduce the problem, stop the EQ session. Compress the recorded packet stream (preferably with bzip2) and e-mail it to me at [email protected] (mailto:[email protected]). At which point I should be able to reproduce the problem.

All information in the packet recording will be maintained in the strictest confidence.

Thanks and Enjoy,
Zaphod (dohpaZ)

[LordCrush]

Zaphod,

i will do this tomorrow .. on raid atm :p it takes normaly 1 respawn to show up the error

/Wave

-- LC

[Alfred]

This might be an interesting piece to the puzzle. (I hope it isn't old news...)

Let me describe what happens normally.

I login to EQ, wait for the character selection screen, start showeq, select my character, show eq works like a champ...

Then as new character(s) and mob(s) show up (no matter what zone), they show up as unknowns. If I want to see everything again, all I have to do is zone and then zone back.



This has been standard procedure for about three weeks for me. I've used myseq but it crashes on the ldon zones.

Then something interesting happened.

I started screwing around with the caller quest in fungus grove. When i first entered the zone, everything was as before, new mobs show up as unknowns, etc. But as I've been learning the do's and don'ts of the caller quest I’ve been doing a lot of dying and rezing. The first few times I’ve been into the zone, I've been completely nude and amazingly... all the NEW MOBS SHOW UP.... This has happened at least three times for me.

Leads me to believe that something I am wearing, or special handshaking that is going on while nude is allowing showeq to continue handling new mobs.

Very interesting indeed.

Now I haven't had a chance to verify whether the old behavior starts once I have all my equipment... (I’ll do that the next time... probably in three or four days)

Any thoughts on this?

[LordCrush]

@Zaphod ... i have send you an email with a packetcapture

@Alfred : i have tried to get the error with a naked lvl 1 ... no err after 10 Mins normally it shows up after 1 or 2 Mins

seems that you are on the right track ... will check further

[S_B_R]

I've had some experience with gear causing weird problems in-game and/or with the login process... Not saying I'm sure this issue is related, but it wouldn't surprise me.

[Raistlin]

Couple of days ago, in PoStorm tracking for Daiku. All of a sudden I notice that there's a POTLOAD of unknown up...and I assumed it was what's described here...every mob that spawned since I zoned in was appearing again as an Unknown.

I'm fairly sure that none of my network equipment is faulty (This is the ONLY problem i've had on any of my computers on my network).

Network setup: DSL going into a NetGear 10/100 DSL router. Netgear connected to a DLink 10/100 hub. Linux box and 2 EQ boxes on the hub (Several other inactive lines connected to the Hub and several active lines connected to the NetGear.)

I've only noticed this since upgrading to .12 (was running .9 for quite some time) to correct the LDON issues, and of course, this is not happening in the LDON zones (Duh, no respawn). Since I generally only use SEQ for offscrean maps and pulling LDON now adays, I havn't noticed unknowns spawning in any zones other than PoStorm that one time.

[Zaphod]

I believe this issue to be fixed in the 4.3.13 CVS commit that I just made (see this thread (http://seq.sourceforge.net/forums/showthread.php?s=&threadid=4046). Thanks and and appropriate props go out to LordCrush who provided me with the packet recording that allowed me to finally diagnose and fix this problem (well, at least this one anyway).

This problem stemmed from one of the unknown bits in the flagsLo part of the packet header being set and including and extra 2 octets before the arq, and thereby throwing a bunch of stuff off. It looks like this bit indicates a NAK (Negative Acknowledgement) being recieved with the extra data being the client arq that wasn't received. This typically would happen if your connection was running into congestion or packet corruption of some sort between your computer and the SOE's server. This also explains why naked characters didn't see the problem as much (all the item data that isn't being sent down at zone in time).

It wasn't a hardware problem, it wasn't unique to 4.3.9 through 4.3.12, but either SOE put that previously unknown bit in the packet to use, or it is just getting excercised more due to the recent changes SOE has made in the packet payloads (such as the new item packets)..

Thanks and Enjoy,
Zaphod (dohpaZ)

[S_B_R]

Thanks Zaphod, and LordCrush! Great news and Great work! :D

[Hendrix_Morton]

Awesome...Played for about 4 hours last night (first time in a month or more I had 4 hours to do anything!) and not a single Unknown in many zones....

Very Nicely Done, Thanks!

[wiz60]

Back when I initiated this thread I was concerned that a fix was unlikely.

I had a chance to test 4.3.13a last night. It works great.

Thanks for all the great work and support.

*dons his cloak of invisibility - slips back into the shadows*

[BlueAdept]

Unfortunately I got quite a few unknowns tonight in Dreadlands. Also some of the mobs didnt seem to be updating. I would go to a blue point, and it was still on the map, but nothing there. I was in the zone for about 2 hours. It wasnt until I was in the zone for about 45 minutes that I started getting the unknowns.

Scrolling back I got the:

Lost sync, relog or zone ot reset
uncompress failed on 0x7201: -3 - data error
no further attempts will be made until zone for direction 2.

Yes it says direction 2.

Then started to get several INVALID PACKET: Bad CRC32's and a bunch of unknowns.

Using the latest/greatest

Also noticed that MakeDropCode (00f3) (dataLen:2 != sizeof (MakeDropStruct):94)

[fester]

Originally posted by BlueAdept
MakeDropCode (00f3) (dataLen:2 != sizeof (MakeDropStruct):94)

This makes me think MakeDropCode opcode listed in opcodes.h is actually the door open opcode (dunno if it is in 4.3.13) but it is the only (or used to be) 2 byte payload opcode and is issued whenever someone clicks on a door or portal (the two bytes is the door id number.)

[BadgerOne]

/bow to Zaphod

works great now. Thanks a bunch guys.

[BlueAdept]

Originally posted by fester
This makes me think MakeDropCode opcode listed in opcodes.h is actually the door open opcode (dunno if it is in 4.3.13) but it is the only (or used to be) 2 byte payload opcode and is issued whenever someone clicks on a door or portal (the two bytes is the door id number.)

I dont think it is a door open opcode. As far as I know there arent any doors in Dreadlands. I believe it happened every time I dropped foraged stuff.

Im using 4.3.13a.

[LordCrush]

Hmm, i play a forage char too ands use /autoinventory with a macro on the run keys, i drop often forged stuff but it did not happen to me yesterday ... i will have an eye on it

[wiz60]

The new version works much better for me than 4.3.10-4.3.12 did. However I tend to stay on the "NPC" filter mode for the spawn list.

A friend pointed out last night that he had lots of strange text on the "ALL" spawn list.

Sure enough when I switched to that mode - I got a lot of nonsense spawns listing. Some had very unusual characters in the name field - unprintables it looked.

I am going to test it a bit more tonight.

I was in a LDON zone at the time - if that helps at all.

[S_B_R]

Last night in an LDON dungeon, I notice that the NPC count in the spawnlist sayed there was something like 1600 npc's in the zone. there were less than 200 mobs (give or take) displayed on the map.... Didn't even dawn on me to switch the spawn filter to ALL....

/shrug

[fester]

Originally posted by BlueAdept
I dont think it is a door open opcode. As far as I know there arent any doors in Dreadlands. I believe it happened every time I dropped foraged stuff.

Im using 4.3.13a.

I looked at this. 00f3 is MakeDropCode but ShowEQ is triggering on the opcode in the wrong direction. 00f3 is a 2 byte opcode from client to server (instructing the server it needs to generate a drop item for the item on the cursor.) 00f3 from server to client is a 94 byte opcode.

It appears ShowEQ is not ignoring the Client->Server direction (as it probably should.) I do see that the item appears in the listing (as a drop item.)

BTW, It would appear that Dreadlands has 7 "doors". But 00f3 is not the door open code. ;-)

[Zaphod]

Originally posted by fester
Unfortunately I got quite a few unknowns tonight in Dreadlands. Also some of the mobs didnt seem to be updating. I would go to a blue point, and it was still on the map, but nothing there. I was in the zone for about 2 hours. It wasnt until I was in the zone for about 45 minutes that I started getting the unknowns.

Scrolling back I got the:

Lost sync, relog or zone ot reset
uncompress failed on 0x7201: -3 - data error
no further attempts will be made until zone for direction 2.

Yes it says direction 2.

Then started to get several INVALID PACKET: Bad CRC32's and a bunch of unknowns.

Using the latest/greatest

Also noticed that MakeDropCode (00f3) (dataLen:2 != sizeof (MakeDropStruct):94)
I don't know what is causing the unknowns for you. I only know that I corrected the situation that was causing many of the unknowns (the ones LordCrush and many others were seeing). If you want to send me a recorded session (--record-file) I'll examine the cause of your situation in the strictest confidence.

As far as the MakeDropCode warning is concerned, it is strictly benign.


Originally posted by fester
This makes me think MakeDropCode opcode listed in opcodes.h is actually the door open opcode (dunno if it is in 4.3.13) but it is the only (or used to be) 2 byte payload opcode and is issued whenever someone clicks on a door or portal (the two bytes is the door id number.)
The DoorClickCode is confirmed as 0x120 and the DoorOpenCode is confirmed as 0x121.


Originally posted by fester
It appears ShowEQ is not ignoring the Client->Server direction (as it probably should.) I do see that the item appears in the listing (as a drop item.)


The only thing that needs to be fixed in regards to the empty payload version of the MakeDropCode packet is its generates of a warning. The false warning will be fixed in my next CVS commit..

Enjoy,
Zaphod (dohpaZ)

[BlueAdept]

So far, that was the only time that I have had it stop decoding. If it happens to me again, Ill start recording the sessions for you.