segfault with gdb backtrace

[sequser246]

Hey guys,

With the latest version (v5.13.5.0) I'm getting the following segfault / backtrace when zoning in from character select. Hoping someone will have some ideas of what I can try! (I've replaced my character's name with MYCHARNAME below...)

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb71d18d0 (LWP 8550)]
0xb751ed2a in strcmp () from /lib/tls/i686/cmov/libc.so.6
(gdb) bt
#0 0xb751ed2a in strcmp () from /lib/tls/i686/cmov/libc.so.6
#1 0x08077cef in SpawnShell::zoneEntry (this=0x83c4818,
data=0x83f52ba "\231Q\216\200MYCHARNAME", len=68) at spawnshell.cpp:663
#2 0x0807871d in SpawnShell::qt_invoke (this=0x83c4818, _id=8, _o=0xbffec90c)
at spawnshell.moc:395
#3 0xb7aba704 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#4 0x08092709 in EQPacketDispatch::signal (this=0x82de2f8,
t0=0x83f52ba "\231Q\216\200MYCHARNAME", t1=68, t2=2 '\002')
at packetinfo.moc:99
#5 0x0808fafb in EQPacketStream::dispatchPacket (this=0x8349a70,
data=0x83f52ba "\231Q\216\200MYCHARNAME", len=68, opCode=8216,
opcodeEntry=0x83c3ce0) at packetstream.cpp:435
#6 0x08090873 in EQPacketStream::processPacket (this=0x8349a70,
packet=@0xbffeca68, isSubpacket=true) at packetstream.cpp:807
#7 0x08090152 in EQPacketStream::processPacket (this=0x8349a70,
packet=@0xbffecb60, isSubpacket=false) at packetstream.cpp:653
#8 0x0809144a in EQPacketStream::handlePacket (this=0x8349a70,
packet=@0xbffecb60) at packetstream.cpp:566
#9 0x0809a97e in EQPacket::dispatchPacket (this=0x83c3140, packet=@0xbffecb60)
at packet.cpp:659
#10 0x0809aacc in EQPacket::dispatchPacket (this=0x83c3140, size=99,
buffer=0xbffecc06 "E") at packet.cpp:583
#11 0x0809c500 in EQPacket::qt_invoke (this=0x83c3140, _id=2, _o=0xbffeec88)
at packet.cpp:400
#12 0xb7aba704 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#13 0xb7abb1e9 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#14 0xb7e4c320 in QTimer::timeout () from /usr/lib/libqt-mt.so.3
#15 0xb7ae1d0e in QTimer::event () from /usr/lib/libqt-mt.so.3
#16 0xb7a4ec36 in QApplication::internalNotify () from /usr/lib/libqt-mt.so.3
#17 0xb7a50a5f in QApplication::notify () from /usr/lib/libqt-mt.so.3
#18 0xb79df28d in QApplication::sendEvent () from /usr/lib/libqt-mt.so.3
#19 0xb7a41b19 in QEventLoop::activateTimers () from /usr/lib/libqt-mt.so.3
#20 0xb79f464b in QEventLoop::processEvents () from /usr/lib/libqt-mt.so.3
#21 0xb7a69f90 in QEventLoop::enterLoop () from /usr/lib/libqt-mt.so.3
#22 0xb7a69c8e in QEventLoop::exec () from /usr/lib/libqt-mt.so.3
#23 0xb7a507df in QApplication::exec () from /usr/lib/libqt-mt.so.3
#24 0x08069af8 in main (argc=Cannot access memory at address 0x0
) at main.cpp:723

Thanks for any help! :)

[ieatacid]

I can't reproduce this crash. Try adding these lines (in blue) in spawnshell.cpp after line 661, to see where it's failing.


Item *item;

if(!spawn->name[0])
seqDebug("SpawnShell::zoneEntry: !spawn->name[0]");
if(!m_player->realName().length())
seqDebug("SpawnShell::zoneEntry: !m_player->realName().length()");

if(!strcmp(spawn->name,m_player->realName()))

[sequser246]

Hi,

Thanks for the reply.



Debug: SpawnShell::zoneEntry: !m_player->realName().length()


The backtrace is somewhat identical so i won't bother reproducing that again!

If you want / are able to give me a list of things to check and trace through the various source files please feel free.

Although I don't have any real experience with C++[1] or the layout / structure of the showeq codebase I'd certainly be able to do some traces and stick some debug lines in if I had a vague idea of how it's all supposed to hang together. :)

Thanks!

[1] My experience comes from other higher level languages such as PHP, python, java, etc...

[ieatacid]

You can remove those line then (marked in red below), and try adding what's in blue


Item *item;

if(!spawn->name[0])
seqDebug("SpawnShell::zoneEntry: !spawn->name[0]");
if(!m_player->realName().length())
seqDebug("SpawnShell::zoneEntry: !m_player->realName().length()");

if(spawn->name[0] && m_player->realName().length() && !strcmp(spawn->name, m_player->realName()))

[sequser246]

Well...

That's got me skittles! :)

You might find this interesting...

When I first zone after running showeq I get:-


Zone: EntryCode: Client
Debug: SpawnShell::zoneEntry: !m_player->realName().length()


For subsequent zoning I just get:-


Zone: EntryCode: Client


...with no Debug line.

[ieatacid]

Could you send me a zone.log from when that happens to [email protected] ? Turn logging on, enter world (or whatever you do to produce this), then shut off logging.

Please empty the file before logging, too, so I don't have to sift through unneeded data.

[purple]

If you're comfortable, you could record yourself in seq and send that to ieatacid to reproduce this. ShowEQ supports reading in tcpdump packets.

Just do:


# tcpdump -i eth0 -s0 -w ibecrashin.log 'udp && 192.168.1.100'


Where eth0 is the device for your ethernet and 192.168.1.100 is the IP of your EQ client. You should start EQ, login, and sit at char select (does char select still exist?), then start the tcpdump. DO NOT START THE TCPDUMP BEFORE YOU START EQ. We do not want your password. We do not need your password. Do not send us your password.

This will make an ibecrashin.log. gzip this so it's smaller:


# gzip ibecrashin.log


this will make ibecrashing.log.gz. Email this to ieatacid (or me, but I'm less likely to help since I haven't played in like 2 years and only poke at seq occasionally) so that he can:



# gzip -d ibecrashin.log.gz
# showeq --playback-tcpdump-filename=ibecrashin.log


This should reproduce for him so he can debug this and fix it.

NOTE: Sending packets dumps like this should only be done if you trust the other end. You are sending your character data, including character name and server.

For more details about tcpdump and playback, you can see this thread:
http://www.showeq.net/forums/showthread.php?t=5263&highlight=playback

[ieatacid]

What a great feature! I had no idea that could be done :)

[sequser246]

I can appreciate how useful the tcpdump would be however I'd like to propose sending the zone.log first (assuming ofcourse that's in a format I can anonymise).

If ieatacid can't see anything glaringly obvious in that then I guess we can move onto the sending of identifiable information... ;)

Will get the zone.log / packet dumps done tonight...

Thanks for the help so far! :)

[purple]

It's going to be a pain in the ass to anonymize the zone.log. It's not gonna be as simple as looking for your player name and finding and replacing. The zone log is going to be hex dumps and you'd need to scrub all the hex and the ascii version. Plus it's gonna have spawn ids all over it and who knows if Sony tracks spawn id logs. It's going to be very daunting if you're not used to looking at it.

If you're paranoid, you should be worried sending someone a zone log. Honestly, the zone log is pretty much the same as the packet log, except the zone log is more human readable. The implementation of the zone log does absolutely nothing to hide who you are. It dumps everything from the data segment for all opcodes sent over the zone server port into the log file as both hex and ascii.

You should do whatever you want. I can totally understand not wanting to send someone a packet dump because of sensitive data in it. But realize that getting a packet dump will make this infinitely easier on the seq developers to fix. In open source projects like these, you're depending on the will of the people who volunteer their time to fix your issue. Whether they have the patience or the drive to do anything about it is up to them.

The EQEmu people used to have a scrubber for packet logs, but I've never used it. I'm not sure how it worked.

[purple]

And obviously no one is asking you post a zone log or packet dump on the forums. You'd email it directly so the only person seeing it is the person you email. I'm sure if you request that it be held in strict confidence ,that that will be obeyed.

[sequser246]

Ok, a pcap file is not going to help you... Every pcap file I've produced that's crashed showeq in realtime has worked fine when getting showeq to playback the pcap file.

Also, I'm getting a different segfault / backtrace now which is kinda odd in itself...



#0 0xb7f21410 in __kernel_vsyscall ()
#1 0xb7457085 in raise () from /lib/tls/i686/cmov/libc.so.6
#2 0xb7458a01 in abort () from /lib/tls/i686/cmov/libc.so.6
#3 0xb7667480 in __gnu_cxx::__verbose_terminate_handler ()
from /usr/lib/libstdc++.so.6
#4 0xb7664d05 in ?? () from /usr/lib/libstdc++.so.6
#5 0xb7664d42 in std::terminate () from /usr/lib/libstdc++.so.6
#6 0xb7664e6a in __cxa_throw () from /usr/lib/libstdc++.so.6
#7 0xb766549e in operator new () from /usr/lib/libstdc++.so.6
#8 0xb766557d in operator new[] () from /usr/lib/libstdc++.so.6
#9 0x0808d78e in EQPacketFragmentSequence::addFragment (this=0x83621e4,
packet=@0xbff32aa0) at packetfragment.cpp:101
#10 0x0809074a in EQPacketStream::processPacket (this=0x8362168,
packet=@0xbff32aa0, isSubpacket=false) at packetstream.cpp:854
#11 0x0809144a in EQPacketStream::handlePacket (this=0x8362168,
packet=@0xbff32aa0) at packetstream.cpp:566
#12 0x0809aa2e in EQPacket::dispatchPacket (this=0x82c9978, packet=@0xbff32aa0)
at packet.cpp:647
#13 0x0809aacc in EQPacket::dispatchPacket (this=0x82c9978, size=328,
buffer=0xbff32b46 "E") at packet.cpp:583
#14 0x0809c500 in EQPacket::qt_invoke (this=0x82c9978, _id=2, _o=0xbff34bc8)
at packet.cpp:400
#15 0xb7a39704 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#16 0xb7a3a1e9 in QObject::activate_signal () from /usr/lib/libqt-mt.so.3
#17 0xb7dcb320 in QTimer::timeout () from /usr/lib/libqt-mt.so.3
#18 0xb7a60d0e in QTimer::event () from /usr/lib/libqt-mt.so.3
#19 0xb79cdc36 in QApplication::internalNotify () from /usr/lib/libqt-mt.so.3
#20 0xb79cfa5f in QApplication::notify () from /usr/lib/libqt-mt.so.3
#21 0xb795e28d in QApplication::sendEvent () from /usr/lib/libqt-mt.so.3
#22 0xb79c0b19 in QEventLoop::activateTimers () from /usr/lib/libqt-mt.so.3
#23 0xb797364b in QEventLoop::processEvents () from /usr/lib/libqt-mt.so.3
#24 0xb79e8f90 in QEventLoop::enterLoop () from /usr/lib/libqt-mt.so.3
#25 0xb79e8c8e in QEventLoop::exec () from /usr/lib/libqt-mt.so.3
#26 0xb79cf7df in QApplication::exec () from /usr/lib/libqt-mt.so.3
#27 0x08069af8 in main (argc=Cannot access memory at address 0x6
) at main.cpp:723
It's just making me think there's something going wrong somewhere with SEQ capturing the packets. I'll get a zone.log mailed to ieatacid since that should be showeq's "understanding" of what it captured...(or as I suspect, didn't capture).

As an aside, ieatacid, when(/if) this turns out to be either a) my fault or; b) my server's fault let me paypal you some beer money for wasting your time... ;-)

[sequser246]

So, um, I've kinda solved this one...

I managed to get it working perfectly with none of the code changes we made to spawnshell.cpp by specifying my EQ client IP address in ShowEQ so that showeq put the IP into its pcap filter.

My guess is all the extra udp traffic on the server was confusing it, somewhat (my first clue was when it did manage to "work" but with incorrect character data - I was level 103 and flagged as a GM).

On the server I also have a bittorrent client which makes use of UDP traffic along with an asterisk server...can you say SIP (UDP)?

Well, now that I've finished wasting everyone's time that beer money offer is still open ;-) (to both purple and ieatacid, infact.)

Any thoughts on having showeq refine its pcap filter on the fly once it detects an EQ client? (If it doesn't do this already?)

[purple]

You should turn on session tracking. It makes the pcap filters lock down a lot more than the default.