Occasional crashing when trying to decode

[xerxes]

I'm occasionally getting a crash when zoning into the world.

Any ideas on this one?



Zone: NewCode: Zone: .
Warning: - OP_Action2 (0x47e9) (dataLen: 4) doesn't match: sizeof(action2Struct):48
Warning: - SEQ: Giving up on finding arq 093d in stream zone-client cache, skipping!
Warning: - Oversized packet fragment requested buffer of size 0 on stream 3 OpCode 0000 seq 0941
Warning: - SEQ: Giving up on finding arq 0944 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0945 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0946 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a06 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a09 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a0a in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a0f in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a10 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a11 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a12 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a13 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a14 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a15 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a16 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a17 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a1a in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a1e in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a2b in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a2d in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a2f in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a34 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a38 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a39 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a3a in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a3e in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a3f in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a40 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a42 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a43 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a44 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a46 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a47 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a48 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a51 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a59 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a5a in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a5b in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a5c in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a5d in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a5e in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a62 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a6b in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a6c in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a6d in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a6e in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a71 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a73 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a74 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a77 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a7a in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a7b in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a7c in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a7d in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a7e in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a7f in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a84 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a85 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a86 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a87 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a88 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a89 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a8c in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a8d in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a8e in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a8f in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a90 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a91 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a93 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a94 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a95 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a96 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a9c in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a9d in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0a9f in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa0 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa1 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa2 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa3 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa4 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa6 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa7 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa8 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aa9 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aaa in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aab in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aac in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aad in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0ab0 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0ab1 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0abc in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0abe in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0ac9 in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0aca in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0acb in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0ace in stream zone-client cache, skipping!
Warning: - SEQ: Giving up on finding arq 0acf in stream zone-client cache, skipping!
Warning: - !!!! EQPacketFragmentSequence::addFragment(): buffer overflow adding in new fragment to buffer with seq 0d71 on stream 3, opcode 0000. Buffer is size 438992 and has been filled up to 438851, but tried to add 505 more!

[cn187]

See my answer here - https://www.showeq.net/forums/showthread.php?8016-Crashing-since-the-last-patch-and-erratic-behavior

[Newby]

I'm curious if these issues are caused by people running showeq on raspberry pi models 1, 2 or 3. If your everquest machine has a gigabit ethernet interface (very common now), those older raspberry pis aren't going to be able to keep up. The 1 and 2 don't have gigabit ethernet, the 3 does but it's routed through the USB bus and that's too slow for true gigabit ethernet.

I don't run showeq on a raspberry pi, but I'd think you'd need at least a raspberry pi 4 or above to handle the traffic.

[xerxes]

I've upgraded the machine I am using since my last post, and simplified my network setup... it doesn't happen often, just occasionally.

I'm not running on a rpi. It was previously a VM and now it's a LXC container. Host is a 12th gen nuc with 32gb of ram with a gigabit nic.

[cn187]

So... I think there's definitely something else going on with this.

I've started getting this pretty frequently myself, and I've never had issues with this on my system prior. (I've already tuned all my kernel params)

Running tcpdump at the same time on the SEQ host show 0 packets dropped, and if I play the same capture back in SEQ and everything works normally with no crash.

So either SEQ's capture is dropping packets independently of the kernel, or there's an issue with the packets getting from the capture thread to the packet processing routine.

The comments in the source talk about a race condition between the capture thread and the main thread and that there's a problem if there's a delay in the capture. So that may be what's happening here.

I don't have an answer yet, but I wanted to let you know that it's definitely on my radar now.

[cn187]

Turning on Real Time Thread under Network definitely seems to help with this - mine had inadvertently gotten turned off, and since turning it back on, I've haven't had any issues.


That said, I'm still going to see if there's anything I can do to help with it in the general case. Though it's likely to be part of a longer/larger refactor.

[cn187]

OK, so realtime definitely helps, though it's not 100% a fix.

Obviously the kernel parameter tweaks have already been covered.

There are only so many levers to pull on outside of that, but there are a couple of pcap options we can tweak that could help.

Snapshot length was set to BUFSIZ which on my system is 8192. And pcap buffer size was not specified, but as far as I can tell the default is 2MB in recent versions of libpcap.

I've reduced the default snaplen to 1k, since the EQ protocol currently limits individual UDP packets to 512 bytes of EQ-related payload + ethernet/ip/udp headers. (~554 bytes). The snaplen is now configurable in case we need to increase it in the future, but that shouldn't be necessary unless Darkpaw changes the underlying protocol. The pcap buffer size still defaults to 2MB, but is now configurable in 1MB increments up to 128MB.

Of course, if, for whatever reason, the system can't process packets fast enough, the buffer will still fill up, packets will still get dropped, and SEQ will still complain and exit.

To that end, I've also added checks so that if PCAP detects that packets have been dropped (which appears to be platform implementation dependent and not 100% accurate), SEQ will report if PCAP thinks they were dropped at the interface, in which case the kernel params should be adjusted; or due to PCAP's buffer being full, in which case, the snaplen/buffersize should be adjusted.

The changes are currently on my dev branch should someone want to test, but I'm hoping to include these in an upcoming bugfix release soonish.

[BlueAdept]

You have been very busy cn187. Dont burn yourself out.