Here is the bazaartracker code

[bazaartracker]

Hi Kids, here is the bazaar tracking program
I wrote about earlier.

LINUX ONLY.
Only source code is distributed.
libpcap required.
NO WARRANTY.
USE MAY CAUSE YOUR ACCOUNT TO BE
TERMINATED blah blah blah.

This is a tracker only...it relies on user-driven
bazaar queries. It is not a spoofer or a money
maker. I mostly wrote it to see how the economy
was behaving...

Hope it works for you as it does for me...if not,
post here and I will try to help.

If people find it useful, someone could incorporate
it into showeq (though I personally do not run
showeq), or start a new sourceforge project
on my behalf.
Feel free to post improvements or bugfixes here.

You can download it here:
http://seq.sourceforge.net/attachment.php?s=&attachmentid=148

[bazaartracker]

Read the comments at the top of the c file
for how to build, use, etc.

[bazaartracker]

well, I attached a gzipped c file to the post but I don't see it anywhere. Broken vboard, or am I just blind?

[bazaartracker]

here it is, maybe

This URL should work:
http://seq.sourceforge.net/attachment.php?s=&attachmentid=148

if not, if you're willing to host the code or open
a sourceforge for it, post here.

[BlueAdept]

If you need a host, I can host it on my site if you want. I havent had time to look at it yet, after I do, Ill put it up at http://www.seqstuff.cjb.net If you want to SF it, that would probably be better. That way you might get more input on it.

[la_large_rat_00]

Very very cool program. Running into a few snags though. When I search the bazaar, the data shows up on my linux box, but it seems to be jarbled.
Search seen: "blue diamond" (user 0 mask:ffffffffffffffffffff)
*** Price change Blue Diamond 1702194242: 1634288 -> 935
*** Price change Blue Diamond 1702194242: 1634288 -> 995
*** Price change Blue Diamond 1702194242: 1634288 -> 1000
*** Price change Blue Diamond 1702194242: 1634288 -> 900
*** Price change Blue Diamond 1702194242: 1634288 -> 800
*** Price change Blue Diamond 1702194242: 1634288 -> 1200
Query done, saw 25 items, time Wed Nov 20 16:52:54 2002

After I search it doesnt write to btrack.dat. When I quit the program it says:

Writing output file.
Wrote 0 items.
You did 1 queries and saw 25 listings

For some reason it's unable to write/doesnt know what to write to btrack.dat.
I would really love to make use of your program. :)

Any Ideas?

[bazaartracker]

Hi, thanks for trying it. Sorry it didn't work for you.

It's not writing a file because it's getting confused about the seller code in each packet...this is also why you are getting all those price change messages.

Your packet format is apparently different from mine. If you'd like to help, you could add a call to dumppacket(data, 0x50) right before the call to addlisting() and send me the output privately.
Or an ethereal/tcpdump capture of a bazaar search...

Has anyone had better success running the program?

[la_large_rat_00]

PMed you, will continue to tinker with it and see if I get anything in the meantime.

[Xipper]

Hi!
Just a note that this worked fine for me! Being a linux newbie I was fearfull that I would have to install a bunch of stuff. When I did a search for pcap, I found the info on where to get it from. When I looked on my local machine in the position it comes from in the command line to compile, I found that I already had it. So I decided to just use what was already here (I could always go back and add the other if it didnt work). Worked fine!

I am running 7.3 redhat.

** might it be possible to have the program show low, high, median, and maybe the WideSpread message in the output file (maybe as an option). I am learning c but am still lacking in knowledge.

Thank you friends!

Xipper Faeriethumper

[Pokesfan]

Before I ask my question, let me make sure you know that I'm very ignorant when it comes to the linux world. I'm learning, but not nearly as fast as I would like.

OK. When I attempt to compile btrack, I get a ton of lines like
"btrack.c:67: stray '366' in program

I'm sure this has to do with my compiler or how I'm set up or something that most linux people wouldn't have any problems with. I've upgraded my libpcap rpm to libpcap-0.6.2-11.7.2.0.i386.rpm (or at least I downloaded and ran the rpm with the -U flag).

What I've done is:
made a btrack directory under root (I know that this isn't "proper" but I'm more than happy always being root and will live with any consequences) put the btrack.c file in that directory.
At a terminal window I went to that directory and typed in
gcc btrack.c -I/usr/include/pcap -lpcap -o btrack

I get pages and pages of those stray errors, then when I'm done, I have nothing in my folder other than my original btrack.c file. Any ideas?

Pokesfan

[Cryonic]

do you have the devel headers for pcap?

[Pokesfan]

Remember, I'm still very ignorant in this. I do not know what you are asking nor how to go about getting this information to post.

I'm going to start a search on this and see if I can determine it, but if anybody can help, I would appreciate it greatly.

Pokesfan

[Poncho]

/bump

I know there are many priorities over and above this, but this would most certainly be a great addition to an already great program. There are some neat possibilities here. Any other thoughts on this?

Poncho "IC Skittles"

[domesticbeer]

Bump

downloaded it this weekend worked great just no writing of the file. Will add the code to dump out hte info and PM you with it later.

[xann]

Hi guys,

Very interesting util, but I am having some problems running it.

I compiled the code without any problems following the instructions in the source.

When I run a query, this is the output I am getting:

Before I ran this I did a 'touch btrack.dat' to create a dummy file.

#./btrack
Done reading saved entries.
Now capturing bazaar queries. Type 'h' for help.

Search seen: "reaper" (user 0 mask:ffffffffffffffffffff)
Segmentation fault
#

(i might have missed an f or two, I am retypign this by hand :) )

Any thoughts?

[Cryonic]

does it SegFault if you delete the .dat file?

[xann]

yeah it does... that's why i created it actually.. before I did the touch on the file (so no file existed) it was still giving me the segmentation fault.

[JoeBloe]

Wow, I was really excited when I saw this.

Has anyone figured out the segfault?

I noticed it was not creating the .dat file so I created one and chmod 777'd it but it is still seg faulting.

Search seen: "" (user 0 mask:e0106001000700ffff)
Segmentation fault

It appears it is catching a null?

RH8.0 stock with full install.

I can't wait to put it to use.

[Undertaker]

Will this not work at the same time as running SEQ? I compiled fine, but when try to run it, get a 'error opening trace interface eth0' message (eth0 being the device that SEQ is sniffing packets on).

Maybe possible to run this at the same time as SEQ, but on a different device (eth1)? Would like to try this, but having a hard time getting second NIC card to play well with the linux box.

[PriestOfDiscord]

Undertaker,

You get this message when the program is not run as root or possibly your network card is not on eth0.

On another note:
I have made a slight modification to code that inserts the info into a mysql database. It basically works at this point, but there are several things that still need work. I haven't coded in C in ages, so it is going slower than I would have liked. I have been spoiled by perl text manipulation.

Anybody else interested in such a modification?

[Undertaker]

Ah Ha! Could have sworn I was on as root, but guess not. Working fine now. Thanks!

[PriestOfDiscord]

hit back button by accident.

ignore

[JoeBloe]

OK, I'm desperate.

I have tried two different versions of libpcap aside from the one that came with RH8.0. But I think I was going at it wrong. I don't appear to have a problem gettiong packets since it shows what I searched for.

so I am trying to run it in gdb now. Can anybody help me with why I can't compile this source file with the -g flag?

Here is the output I get.

Done reading 0 saved entries.
Now capturing bazaar queries. Type 'h' for help.

Search seen: "reaver" (user 0 mask:ffffffffffffffffffff)

Program received signal SIGSEGV, Segmentation fault.
0x08048b4c in addlisting ()
(gdb)

[JoeBloe]

This is the line causing the segfault.

if (seller->price != price) {

I put a printf between every line in the area and I get the printf before this line but not after.

I will try to get gdb to work but it'll be a learning curve.

[JoeBloe]

Geez, got the -g option to work. Had to put it before the source file.

---------


(gdb) run
Starting program: /root/tracker/btrack
Done reading 0 saved entries.
Now capturing bazaar queries. Type 'h' for help.

Search seen: "reaver" (user 0 mask:ffffffffffffffffffff)
adding Reaver(1) 7000000
1
2

Program received signal SIGSEGV, Segmentation fault.
0x08048b86 in addlisting (itemcode=29794, itemname=0xbffff5e0 "Reaver(1)",
sellercode=26634, price=7000000) at btrack.c:278
278 if (seller->price != price) {
(gdb) list
273 additemctr++;
274 printf("1\n");
275
276 if (seller = searchseller(item, sellercode)) {
277 printf("2\n");
278 if (seller->price != price) {
279 printf("here?\n");
280 printf("*** Price change %s %d: %d -> %d\n", item->name,
281 seller->sellercode, seller->price/1000, price/1000);
282 seller->lasttime = seller->chgtime = curtime;

[JoeBloe]

This fixes the segfault. It is caused by an unchecked pointer.

I am now running. Thanks for the util!


253,255c253,257
<
< for (sc = item->sellerlist ; sc ; sc = sc->next) {
< if (sc->sellercode == sellercode)
---
> if (item && item->sellerlist)
> {
> for (sc = item->sellerlist ; sc ; sc = sc->next)
> {
> if (sc->sellercode == sellercode)
257c259,261
< }
---
> }
> }
> return NULL;
276c280
< if (seller->price != price) {
---
> if (seller && seller->price != price) {

[xann]

Hi guys, a friend of mine helped me fix this problem, he figured out the same solution to the problem, returning the null code. here's a copy of my source changes:

----- old source code ------

searchseller(item_t *item, unit4 sellercode)
{
seller_t *sc;

for (sc = item->sellerlist ; sc ; sc = sc->next) {
if (sc->sellercode == sellercode)
return sc;
}
}

----- new source code ------

searchseller(item_t *item, unit4 sellercode)
{
seller_t *sc;

for (sc = item->sellerlist ; sc ; sc = sc->next) {
if (sc->sellercode == sellercode)
return sc;
}
return NULL;
}



On another note, I added what I think is a nice mod to make things a little clearer when you get the list printed out.. This is simple stuff I recalled from my old days of running a BBS and using ANSI color codes:

----- old source code ------

printf("BELOW SAVED LOW");

----- new source code -----

printf("\e[31mBELOW SAVED AVERAGE (%dpp vs %dpp)\e[0m\n", currentlow->price/1000, item->savedlow/1000);

----------------------------------

This outputs at the end of a line with a low price hit text that looks something like this:

BELOW SAVED AVERAGE (5900pp vs 6350pp)

I really like this a lot more for the output, it lets you quickly identify possible interesting items, highlights them in red text if your terminal supports ANSI color codes and summarizes the current low price versus whats being stored in the .dat file.

Also, I changed the text because the saved price appears to actually be an average price over the # of samples, rather then the actually lowest price you have ever seen.. Makes a little more sense to me this way.

See ya,

Xann

PS: Thanks for this great little piece of code, I am having a blast with playing with it.

[BlueMonk]

Im trying to build a database of prices and traders on my server. I would like to know which traders usually sell for the lowest prices.

In order to do that, I would need the trader name, instead of its code, which changes from section to section.

Anyone have any idea how I can capture the trader name? I guess it doesnt come inside the bazaar packet, but im not sure about that since Im a newbie in packet sniffing.

Thank you.

BlueMonk

[BlueMonk]

Hello there,

I loved your little program btrack. I modified it and built a SQL database with all the items, prices and traders being offered at the bazaar. Using that DB, I am able to know almost exactly what is the actual selling price of every item, since I update the prices 3 or 4 times a day.

The problem is that with the patch of 1-9-2003. your code stopped working. I tried to figure out how it captures bazaar packets, but Im at a loss at finding the problem.

Arent the packets encripted? Your program doesnt seem to use any key for decripting the packets. If you can help me to fix it, I would be greatly grateful. Could you explain me how your code works? Especifically, the handler function, where it gets the opcode from, what is the eqver, etc.

Any help would be appreciated.

Thank you,

BlueMonk.

[spungee]

just downloaded and compiled it, but it doesn't see any packages

[Nurseling]

Just curious does any one have working code for this?

[tamasine]

i have a set of patches to showeq (when it works) that dump bazaar packets, and a perl scripts to mungle the results. unfortunately they're not really up to the standard of the rest of the code.. so i was trying to clean it up before posting.
t.

[papiraul]

Would be awsome to see that code posted if ya get it working after this f..ed up patch SoE made the last time..!!
Keep it up man!!

[NSParadox]

Wow,

I just randomly found ShowEQ looking for a tool that does exactly this (track bazaar stuff).

I'm also looking for a good open source project.

Where is everyone's patches? Are you going to be putting the code w/ patches that have been submitted to you? I'd be willing to help code btw (although I don't know what needs to be implemented/fixed... haven't tried it).

[Fenceman]

NS,

I couldn't tell you where to begin, but I know this used to be a tool seperate from SEQ that worked pretty well with a few tweaks.

Perhaps the best thing to do at this point is wait until there is a working version of SEQ and then see if tamasine has further plans of releasing a patch to do something with the bazaar packets.

Other than that, you have the option of getting your hands dirty with some coding. As it doesn't sound like you know much of SEQ, you may want to read the posts in the development forum from around the time of PoP release forward. Packets are sent from the EQ server encrypted and compressed, so there's some work ahead of you if you want to try this on your own.

At one point, I had started to take the original program (thanks bazaartracker) and port it to Perl--which I am more comfortable workin with, but as it turns out, I'm not smart enough to figure this whole packet thing out and they proved fatal to my attempts :(

So, personally, I'm hoping for tamasine or someone else to release something as an add-on of sorts for SEQ. That way SEQ can handle all the decompression/decryption.

Note: I'm a long-time lurker and user of SEQ, but not an active participant, so this is knowledge I've picked up through observation and is at the mercy of my own misunderstanding :)

[NSParadox]

Hrm, I thought ShowEQ was currently working (I looked through messages about ShowEQ working on the same box as winex running EQ, and these messages were recently posted). I'm a dual booter, so I don't have a dedicated machine to try ShowEQ out with, but I thought they had broken the latest encryption scheme.

Well, if it's not yet broken, I'd be more than happy to contribute some code as soon as its done. :)

[NSParadox]

OK, I read up on the dev boards... not so good to only find threads using search.

I'll patiently await the day ShowEQ works again.

:)

[Fenceman]

Heh, fixed/broken is a matter of high debate atm. There are some who have struck out on their own and have a working version of SEQ (or, something like SEQ which isn't actually SEQ, if you will). And some believe that the devs have a working version and are withholding it :rolleyes:

Whatever the case, SEQ is not, at this time, working for the 'masses'

As for recent posts, keep in mind that each time Sony sends you a new eqgame.exe, something changes which could mean a little or a lot of work for SEQ devs. So, it's possible that the message you saw was during a 'working' time period and there was a sebsequent change which broke SEQ again.

cheers

[Raistlin]

BTracker.

What a SWEET little piece of code this is. And an awesome idea at that...my hat off to you for this.

I'm assuming no-one has this working at this point due to the network code changes that have plagued SEQ recently. I'm sitting here trying to figure out if this could be added into SEQ as a "tool" so to speak, almost as a subsection of SEQ...the thing is it's either added to SEQ or you're going to have to re-design the packet thing every time the network protocol changes (which seems to happen regularly now adays). Does anyone know if the issue is that the bazaar packets are encrypted at this point or is it just the opcode compression/grouping that's causing the problem (not that this is small, but it's not as hard as breaking an encryption scheme)?

Anyway, i'll have to look into integrating this with SEQ in some wway at some point...

[tamasine]

I *had* bazaar packets decoded.

They were just mangled - no encryption, but there is (i think) compression. I got the packets after SEQ had demangled/decompressed/op-code id'd them.
As i haven't touched C++ for so long, the output was limited to sprintf's (none of this wierd emitting stuff) which i piped out into a perl script that did the rest of what btracker.c did (db maintainance/price alerting). It could be rather neat to fold that into SEQ, but perl is my language of choice for playing with data.

The bazaar response struct was basically just seller-id/price/object-id/name (with padding) and thus rather simple.. but only after it'd been through SEQ.
tam

[icyman]

Ok, I was really bored waiting 8 hours(!) for a petition reply, so I started hacking around with this some.

It appears the new Bazaar search results opcode is 0x0209.

Not sure if the packet layout has changed, that's where I am now.

Here's one of the packets I recorded:



What How much Who
Artisan's Seal(1) 6000pp ???
Artisan's Seal(1) 6000pp ???
Artisan's Seal(1) 6000pp ???

BazaarSearchResult: 0209 len 266 [Server --> Client] ID:7
000 | 09 02 07 00 00 00 ## ## 00 00 oo oo 00 00 pp pp | ..........9\..f.
016 | 00 00 xx xx xx xx 00 00 00 00 41 72 74 69 73 61 | ....[.....Artisa
032 | 6e 27 73 20 53 65 61 6c 28 31 29 00 00 00 00 00 | n's Seal(1).....
048 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
064 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
080 | 00 00 00 00 00 00 00 00 00 00 07 00 00 00 01 00 | ................
096 | 00 00 39 5c 00 00 pp pp 00 00 80 8d 5b 00 00 00 | ..9\..f.....[...
112 | 00 00 41 72 74 69 73 61 6e 27 73 20 53 65 61 6c | ..Artisan's Seal
128 | 28 31 29 00 00 00 00 00 00 00 00 00 00 00 00 00 | (1).............
144 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
160 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
176 | 00 00 07 00 00 00 01 00 00 00 39 5c 00 00 pp pp | ..........9\..f.
192 | 00 00 80 8d 5b 00 00 00 00 00 41 72 74 69 73 61 | ....[.....Artisa
208 | 6e 27 73 20 53 65 61 6c 28 31 29 00 00 00 00 00 | n's Seal(1).....
224 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
240 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
256 | 00 00 00 00 00 00 00 00 00 00 | ..........

Legend:
## ## amount of an item for sale (little endian)
oo oo object id (little endian, verified using Magelo)
pp pp player id (little endian) (all replaced, I am paranoid)
xx xx xx xx price of the item, in cp (little endian)

Each item appears to start with 0x07, and is 58 bytes long.



Hope this helps someone. I'm at the point of dumping out the entire contents of the data parameter handed in to the handler function, just to see where the offsets are that I should be working with. (I have no clue what index 42 is, for example, which appears to be a (big endian) version of the opcode (at least, in previous versions)).


][cyman

[tamasine]

heh.

the packet structure of bazaar responses has totally changed:

previously it was count, cItemInShop[count]. now there's no count, and a rather bare item struct.
annoyingly, the item-on-corpse packet seems now devoid of item information which suggests to me that this has been shuffled into the body of eqgame.exe (ie it just sends item numbers now). most annoying ;)

if anyone wants, and i remember - i've a patch that'll put stdout reporting of shop items/bazaar queries, and a perl program that takes these in and thinks about it (with beep warnings for !buyme! deals)

but it could really do with being properly integrated into this wierd emit thang that seems to go on.

[icyman]

Sounds cool tamasine, I for one, would love to look over the code. I'm not sure where I am heading with this yet, but I am thinking about setting up some scripting (somehow, this is totally an off-the-top-of-my-head thing atm) to watch bazaar prices of various things, or all things, or something specific, and populate a web page or save to a database, or something, to take advantage of this information. Email alerts, etc, etc would be rally nice. (I guess the trick would be to figure out how I would take advantage of this info from work, which is behind a firewall, which I can tunnel out of, but only to an SSH account on my trusty linux box ;-).

Hmm - WineX + X11 forwarding through SSH, through the SSL tunnel.... ick (but interesting) ;-)


][cyman

[Chaz]

Originally posted by tamasine
heh.

if anyone wants, and i remember - i've a patch that'll put stdout reporting of shop items/bazaar queries, and a perl program that takes these in and thinks about it (with beep warnings for !buyme! deals)


Yes I also would like this posted please.

[tamasine]

it's quite distrubingly simplistic.. honest!

and icyman.. your problem is going to be triggering your eqclient to run bazaar searches. (though you could leave the cursor in the search string entry, and push keystrokes into it - with enter firing the search)

ok. i'll try and remember when i get home ;)

[icyman]

Yeah, that is going to be the trick. I'd like to avoid running something on the windoze box, so I'm not too sure how I am going to swing this.

It will need some thought I suspect. :)


][cyman

[tamasine]

ok.. don't say i didn't warn you about hackyness:

patch showeq. run with:
$ showeq | ./thing.pl

(attachments. blah)

Index: src/everquest.h
================================================== =================
RCS file: /cvsroot/seq/showeq/src/everquest.h,v
retrieving revision 1.55
diff -c -r1.55 everquest.h
*** src/everquest.h 20 Apr 2003 07:01:30 -0000 1.55
--- src/everquest.h 24 Apr 2003 17:08:23 -0000
***************
*** 1,3 ****
--- 1,4 ----
+ #define NEWITEMSTYLE
/*
* everquest.h
*
***************
*** 212,217 ****
--- 213,229 ----
/*0196*/ uint8_t unknown0196[40]; // ***Placeholder
};

+ struct myitemStruct
+ {
+ /*0000*/ char idfile[30];
+ /*0054*/ char name[64]; // name
+ /*0064*/ char lore[80];
+ /*0174*/ uint8_t unknown1[200-174];
+ /*0200*/ uint32_t cost;
+ /*0204*/ uint16_t id;
+ /*0206*/ uint8_t unknown[460-206];
+ };
+
// Common Item Structure (non-book non-containers)
struct itemItemStruct : public itemStruct
{
***************
*** 1916,1921 ****
--- 1928,1949 ----
struct keyStruct
{
/*0002*/ uint64_t key;
+ };
+
+ struct bazaarResult
+ {
+ uint16_t unknown1;
+ uint16_t unknown2;
+ uint16_t unknown3;
+ uint16_t unknown4;
+ uint16_t itemId;
+ uint16_t unknown5;
+ uint16_t sellerId;
+ uint16_t unknown6;
+ uint32_t price;
+ uint16_t unknown7;
+ uint16_t unknown8;
+ char name[64];
};

// Restore structure packing to default
Index: src/opcodes.h
================================================== =================
RCS file: /cvsroot/seq/showeq/src/opcodes.h,v
retrieving revision 1.26
diff -c -r1.26 opcodes.h
*** src/opcodes.h 21 Apr 2003 05:01:29 -0000 1.26
--- src/opcodes.h 24 Apr 2003 17:08:24 -0000
***************
*** 230,233 ****
--- 230,235 ----

#define cChatFiltersCode 0xff41

+ #define BazaarResultCode 0x0209
+
#endif // OPCODES_H
Index: src/packet.cpp
================================================== =================
RCS file: /cvsroot/seq/showeq/src/packet.cpp,v
retrieving revision 1.51
diff -c -r1.51 packet.cpp
*** src/packet.cpp 19 Apr 2003 03:23:03 -0000 1.51
--- src/packet.cpp 24 Apr 2003 17:08:37 -0000
***************
*** 1945,1977 ****
{
cItemInShopStruct *citems;
citems = (cItemInShopStruct *)(data);
!
! if (citems->count)
! {
! // Determine the size of a single structure in
! // the compressed packet
!
! int nPacketSize=((len - 4)/citems->count);
!
! // See if it is the size that we expect
!
! int nVerifySize = sizeof(itemInShopStruct);
!
! #ifdef PACKET_PAYLOAD_SIZE_DIAG
! if (nVerifySize != nPacketSize)
! {
! printf("WARNING: cItemInShopCode: packetSize:%d != "
! "sizeof(itemInShopStruct):%d!\n",
! nPacketSize, nVerifySize);
! unk = true;
! }
! #endif
! for (int i=0; i<citems->count; i++)
! emit itemShop((const itemInShopStruct*)&citems->compressedData[i*nPacketSize], nPacketSize, dir);
}
!
break;
} /* end ItemInShopCode */

case MoneyOnCorpseCode:
{
--- 1945,1992 ----
{
cItemInShopStruct *citems;
citems = (cItemInShopStruct *)(data);
! data+=6;
! len-=6;
! int count;
! count=len/(sizeof(myitemStruct));
! printf("got a shop (%d) %d %d\n",count,len,sizeof(myitemStruct));
! logData("/tmp/shop",len,data);
! myitemStruct *fish=(myitemStruct *)data;
! for (int i=0; i<count; i++) {
! fish=(myitemStruct *)(data+(sizeof(myitemStruct)*i));
! printf("item: %x %d %s\n",fish->id,fish->cost,fish->name);
! fflush(0);
}
! // for (int i=0; i<citems->count; i++)
! // emit itemShop((const itemInShopStruct*)&citems->compressedData[i*nPacketSize], nPacketSize, dir);
! // }
!
break;
} /* end ItemInShopCode */
+
+ case BazaarResultCode:
+ {
+ int numb;
+ // printf("fooop!\n");
+ // logData("/tmp/bz",len,data);
+ data+=2;
+ len-=2;
+ numb=len/(sizeof(bazaarResult));
+ struct bazaarResult *result=(struct bazaarResult *)(data);
+ if (result->sellerId == 0xffff) {
+ printf("BAZAAR: end of query\n");
+ fflush(NULL);
+ break;
+ }
+ // printf("parsing %d items\n",numb);
+ for (int i=0; i<numb; i++) {
+ result=(struct bazaarResult *)(data+(i*sizeof(bazaarResult)));
+ // printf("debug: %d %d %d %d %d %d %d %d\n",result->unknown1,result->unknown2,result->unknown3,result->unknown4,result->unknown5,result->unknown6,result->unknown7,result->unknown8);
+ printf("BAZAAR: %x %x %d %s\n",result->itemId,result->sellerId,result->price,&(result->name));
+ }
+ break;
+ }
+

case MoneyOnCorpseCode:
{





*******************
thing.pl
*******************
#!/usr/bin/perl
$mult=15;
($x)=@ARGV;
@ARGV=();
$zone=$x if ($x);
print "using multiplier $mult\n";
print "zone: $zone\n";
@items=("Iron Oxide", "Sky Stone");
&load_btrack;
&read_wants;
while (<>) {
# s/'/=/g;
&dump_db;
# print;
if (/Zone: Entered: ShortName = '([A-Za-z0-9]+)'/) {
$zone=$1;
print "entering zone $zone\n";
&read_wants;
next;
}
if (/^Item: ([^\(]+)\(([0-9]+)\), Slot:.*Value: (.*)$/) {
$name=$1;$id=$2;$val=$3;
next if ($name =~ /Container:/);
print "item: $name\n";
$silent=1;
&loot_value($name,$val);
$silent=0;
next;
}
if (/SEQClosing/) {
&dump_db;
}
if (/^BAZAAR:/) {
&bazaar($_);
next;
}
if (/^item: ([0-9a-f]+) ([0-9]+) (.*)$/) {
# print;
# print "seen item: $3 id $1 at ".int($2/1000)."pp\n";
&shop_value($3,$2,$1);
next;
}
if (/^Item Shop: '(.+)' \(([0-9]+)\), Value: (.*)/) {
&shop_value($1,$3,$2);
next;
}
# if (/^Item Shop: '(.+)' \([0-9]+\), Value: ([0-9]+) [^P]/) {
# &shop_value($1,0);
# }
if (/^Item: Trade Item \[.*\]: '(.+)' \(([0-9]+)\), Value: (.*)/) {
&loot_value($1,$3,$2);
&dump_db;
next;
}
# if (/^Item Shop: '(.+)' \([0-9]+\), Value: ([0-9]+) Plat/) {
# &shop_item($1,$2);
# next;
# }
# if (/^Item Shop: '(.+)' \([0-9]+\), Value:/) {
# &shop_item($1,1);
# next;
# }
print;
}

sub bazaar {
($line)=@_;
if ($line =~ /end of query/) {
print "End of bazaar reply - $bazitems seen\n";
$bazitems=0;
$dump=1;
return;
}
$bazitems++;
print "starting query\n" if (!$bazitems);
($baz,$id,$seller,$price,@rest)=split(/ /);
open(XX,">>/home/snow/bzrlog");
$time=time;

print XX "$time:$id:$seller:$price:@rest";
close XX;
# price in cp
$pprice=int($price/1000);
$name=join(" ",@rest);
chomp $name;
$name=~ s/\([0-9]+\)$//;
# print "checking $name at $price $buyat{$name} $vendor{$name}\n";
if ($vendor{$id} > $pprice || $buyat{$id} > $pprice) {
print "**Look! $name for $pprice ($vendor{$id} [$buyat{$id}])\n";
}
if (2 * $price < $med{$id}) {
print "Look! $name for $pprice (".int($med{$id}/1000)." is the median bazaar [over $medcount{$id}])\n";
}
$med=($med{$id}*$medcount{$id})+$price;
$medcount{$id}++;
$med{$id}=int($med/$medcount{$id});
# print "Median ($id) -> $med{$id} ($medcount{$id})\n";
if ($lowcount{$id} == -1) {
$lowcount{$id}=0;
$lowmed{$id}=0;
}
if ($price<$lowest{$id} || $lowmed{$id}==0 || !$lowest{$id}) {
$lowest{$id}=$price;
$lowmed=($lowmed{$id}*$lowcount{$id})+$price;
$lowcount{$id}++;
$lowmed{$id}=int($lowmed/$lowcount{$id});
# print "Low median -> $lowmed{$id} ($lowcount{$id})\n";
}
}

sub shop_value {
($item,$value,$id,$ow)=@_;
# open(X,">>shopitems");
# print X "$item : $value\n";
# close X;
# print "in shop: $item $value $id\n";
# $value=&decode_value($value) if (! ($value =~ /^[0-9]+$/));
$value=int($value/1000);
&shop_item($item,$value);
$value=int($value*0.95);
# $id=sprintf("%x",$id);
# if (($zone && $zone ne 'Bazaar') || $ow) {
# print "($zone) setting $item to vendor $value\n" if ($vendor{$item} != $value);
# $vendor{$item}=$value;
# $vendor{$id{$item}}=$value;
# $buyat{$id{$item}}=$value if ($buyat{$id{$item}}>$value);
# $dump=1;
# }
if (!$name{$id}) {
$name{$id}=$item;
$id{$item}=$id;
$vendor{$id}=-1 unless ($vendor{$id});
$dump=1;
print "seen new: $item ($id) [$value]\n";
$med{$id}=0;
$medcount{$id}=0;
$lowmed{$id}=-1;
$lowcount{$id}=-1;
$buyat{$id}=0;
}
}

sub loot_value {
($item,$value,$id)=@_;
# open(X,">>lootitems");
# print X "$item : $value\n";
# close X;
&shop_value($item,$value,$id,$override);
if ($value>1) {
print "VALUABLE: $item ($value pp)\n";
}
}

sub shop_item {
($item,$value)=@_;
# value in pp!
# $item=~ s/^\*//;
$seen_price{$item}.=":$value";
foreach $check (@items) {
if ($item =~ /$check/i) {
print "BUY THIS!: $item\n";
}
}
if ($price{$item}>($value*$mult) && $value !=0) {
print "Check this out.. bazaar price: $price{$item} - here $value : $item\n";
}
# print "Vendor: $vendor{$item} : $value\n";
if ($vendor{$item}>$value) {
print "Vendor price higher: $vendor{$item} - here $value : $item\n";
}
}

sub read_wants {
open (X,"wanted");
@items=();
while ($i=<X>) {
chomp $i;
push(@items,$i);
}
close X;
print "looking for:\n";
foreach $i (@items) {
print "$i\n";
}
}
sub load_btrack {
open (X,"bt.ff");
while (<X>) {
chomp;
($id,$med,$medcount,$lowmed,$lowcount,$vendor,$buy ,@name)=split(/ /);
$name=join(" ",@name);
$name=~ s/^\*//;
$price{$name}=int($med/1000);
$vendor{$name}=$vendor;
$buyat{$name}=$buy;
$id{$name}=$id;
$med{$name}=$med;
$medcount{$name}=$medcount;
$lowmed{$name}=$lowmed;
$lowcount{$name}=$lowcount;
$price{$id}=int($med/1000);
$vendor{$id}=$vendor;
$buyat{$id}=$buy;
$name{$id}=$name;
$med{$id}=$med;
$medcount{$id}=$medcount;
$lowmed{$id}=$lowmed;
$lowcount{$id}=$lowcount;
}
close X;
}

# itemid median med-count median-low low-count vendor-price buy-at name

sub dump_db {
return if (!$dump);
return if (time < $dbtime+20);
$dbtime=time;
print "dumping\n";
$dump=0;
open X,">bt.ff";
foreach $id (sort keys(%name)) {
print X "$id $med{$id} $medcount{$id} $lowmed{$id} $lowcount{$id} $vendor{$id} $buyat{$id} $name{$id}\n";
}
close X;
}

sub decode_value {
($val)=@_;
local ($pp,$gp,$sp,$cp);
chomp $val;
$pp=$1 if ($val =~ /([0-9]+) Platinum/);
$gp=$1 if ($val =~ /([0-9]+) Gold/);
$sp=$1 if ($val =~ /([0-9]+) Silver/);
$cp=$1 if ($val =~ /([0-9]+) Copper/);
$value=$pp*1000+$gp*100+$sp*10+$cp;
# print "$val -> $value\n";
return $value;
}

[Fatal]

Is this still functioning?

[tamasine]

no.
both opcodes have changed, and there were a couple 'o other things that didn't merge right. haven't had time to fix 'em.

i don't expect the structures to change though

[guy smiley]

I actually figured out how to do something useful today, go me.

#define BazaarResultCode 0x0213

[papiraul]

Awsome smiley..
Made the changes to showeq.. compiled it.. Ran it with the pearl script.. Works nicely..

Any chance someone can work this into seq,, so you can open a bazaartracker window and stuff?

[Darbin]

I want to use the original script to dump the searches into a mysql database for later use and searching, however i cant seen to find out how to get the original C script to work with the new opcodes.

Has anyone changed the .c code to work on the recent patch? I've looked for opcodes in the .c file and tried replacing the ones there with the one posted above but it still wont work. Thanks.

[domesticbeer]

void
handler(unsigned char *unusedbuf, struct pcap_pkthdr *hdr, u_char *data)
{
unsigned int shift, t, udplen, opcode, eqver;
unsigned int isquery;
unsigned int itemcode, sellercode;
unsigned int price;
uint4 add_res;
uint4 sellerfilt;
char item[200];
uint4 searchtab[MAX_RESULTS];
t = rdshort(&data[12]);

/* IP packet */
if (t != 0x0800)
return;

/* packet data starts at 42 */
opcode = (int)data[42];
if (opcode != 0x12) {
return;
}

shift = (int)data[43];
if (shift) {
data += shift / 2; /* XXX not sure if this is right */
}
eqver = rdlong(&data[0x32]);
if (eqver == 0x11420c00) {
time_t t = time(NULL);
/* may not really be the end due to reordering */
printf("Query done, saw %d items, time %s\n", itemctr, ctime(&t));
dumpsearch(searchtab, itemctr);
queryctr++;
itemctr = 0;
}
if (eqver != 0x11420700) {
return;
}

isquery = rdshort(&data[0x40]);

{
unsigned int smask1, smask2, smask3;
char search[200];

sellerfilt = rdshort(&data[0x36]);

smask1 = rdlong(&data[0x38]);
smask2 = rdlong(&data[0x3c]);
smask3 = rdshort(&data[0x40]);
rdstring(&data[0x42], search);
if (smask1 == 0xffffffff || smask2 == 0xffffffff || smask3 == 0xffff) {
/* XXX this sucks! I think we need to recognize who is the client
* and who is the server to figure this out correctly...
*/
printf("\nSearch seen: \"%s\" (user %x mask:%x%x%x)\n", search,
sellerfilt, smask1, smask2, smask3);
return;
} else {
/* not a query */
}
}
itemcode = rdshort(&data[0x38]);
sellercode = rdshort(&data[0x3c]);
price = rdlongrev(&data[0x40]);
rdstring(&data[0x48], item);
add_res = addlisting(itemcode, item, sellercode, price);
searchtab[itemctr] = itemcode;
if (itemctr < MAX_RESULTS)
itemctr++;
}


The packet handler in the original code is in here, but I believe this was before the increase in the opcode length. Not sure, but I believe that is why just putting in the new opcode wont work.

[Darbin]

Thanks, i was hoping that it would be as simple as pointing it to the right spots in the packet where the relevant info was.

[tamasine]

the original bazaartracker code expected everything to be plaintext - it broke as soon as compressed/coded packets appeared.
my patches to seq to show bazaar packets only work when seq is running from zone-in and has synched with the packet stream. at any other time, the data just never gets to that bit of code (not sure why/how the uncoding bits work - i just know you need to be snooping from before zonein. this was a 'relatively' recent change though i don't recall when, it was after the initial change that broke bazaartracker)

my first attempt was to understand the seq packet handler, and add that to bazaartracker.. after i gave that up ;) .. i patched the output into seq and wrote a quick db/tracking perl script to do roughly the same thing.

if anyone can break-out the seq packet handler code into a reusable form it would be very useful .. if you did it so it worked like a unix pipe, even better *grin* - i can write perl ;)

[rucool]

Is any of this stuff still working? If the btrack.c is what is the offset needed? Hopefully its that simple.

I have compiled the patch to show eq and ran the thing.pl using

showeq | perl thing.pl

I have had no luck getting this to work. I changed the define bazaarcode to what was posted by Guy Smiley (thanks man) but it didn't work either.

Is no one interested in this anymore? Or does it all work and I am screwing something up somewhere. Any pointers/hints or just plain on kick in the ass to get me going in the right direction will be most greatfully appreciated.

/cheer showeq

Rucool

[tamasine]

don't bother running the perl until you get showeq outputting bazaar info.
don't bother with btrack.c unless you totally re-write it.

i think the updated opcode was the last thing to change. certainly works for me, but i can't check atm.

you *need* to have seq running before you zone into the bazaar. or it won't work.


failing all that, if you're seeing nothing when you do bazaar queries, turn on unknown-packets for a bit, then trawl the logs looking for obvious bazaar responses.. then take their opcode and set that as bazaarresponse.

sorry this isn't more helpful, but since the lack of protocol changes recently it 'just works' for me.

[rucool]

OK. I got the seq part going some what.

#define BazaarResultCode 0x0212

Ran the perl script with showeq..
My only ouput is:

End of Bazaar reply - 0 seen
End of Bazaar reply - 7 seen

Is there something I need to change in thing.pl?

[tamasine]

no. that means it's working perfectly

you did a bazaar query, and got 7 replies back. hopefully the price info for those things was added to its database, but nothing lept out as special (or under the 'buy at' price).

like i said - it's very hacky and extremely personalised. on the whole i used it for checking on specific items (SCHW <3k [at the time]) and for things under the shop buy-at price (which needs manually adding into the database)

now you're getting results into the script you need to modify the perl to do whatever you want from it. it shouldn't be too hard to work out, or just re-write from scratch. i believe the seq output is fairly obvious (something like item id, price (cp), item name) the perl at the moment is rather less sophisticated than the original bazaartracker.c in terms of its averages and skew reporting.

if you open a shop you should also see a spew of information - it'll check for anything in the @items array (iron oxide and skystones at the moment) and alert you to these and also for items where the bazaar price seems higher.

[PriestOfDiscord]

I have just added in these changes with the most recent updates made by codepig. The only update needed to get this working again is to change the BazaarResultCode opcode to:
0x01ec

[monklett]

Patch worked for me too, once I updated it with the opcode from Codepig. The 'thing.pl' perl script is a bit rudimentary, but it works.

I am also exploring the possibility of doing something with a MySQL interface in Python, and my understanding of the bazaar is as follows:

1) the only time bazaar data is sent down the wire is when you use the bazaar search feature, and

2) there is no other information sent to the client (e.g. new trader, updated prices, etc.)

Anyone know if this generally correct or if there any other data available in the stream that is not a result of a bazaar query?

[tamasine]

never seen any.

and the /bazaar search <blah> <blah> functionality i found documented once isn't actually in-game (bazaar searches from command line). so you can't make bazaar hotkeys ;(

if you want to be clever, you could see if the bazaar vendor ids tie to player-ids, and thence add seller-name into the output. that'd be cool *grin*. i never got around to it.

t.

[weawen]

any one know if this works after all the LDoN patches?

[R0x0r_Mc0wnage]

Where are the changes that codepig made? I don't see 'em..

[rampage]

As of November 20 2003 this no longer functions.

I'm not bitching, just letting people know. I'd like to see this work but suck at C but I'll look over it to see what I can do.

[tamasine]

works for me. OP_Bazaar = 01ee

haven't sorted out the shop info yet - that looks to have been utterly redone since LDoN

[Reason]

Maybe I am missing something then. Can someone tell me exactly how to go about applying this patch?

[Glee]

I worked on this a bit today, and I'm attaching a patch. The patch works for me at least, but no guarantees :D It is a modified version of the patch posted by tamasine.

This is my first patch for ShowEQ, so I'm not sure I did it right. Post here if you have any problems. I used the opcode 0x01ec and changed the structure a bit too.

The patch is against current CVS (11/28). I don't think any of the files have changed since 4.3.18 was released, so it should work on the latest tarball.

[R0x0r_Mc0wnage]

Is this for the thing.pl or does it work seperately?

[R0x0r_Mc0wnage]

I got it working, but it's not dumping the database of what it sees? :|

[Glee]

This patch should be used with thing.pl (or some other program that can parse the data it spits out). It's almost identical to the original patch to ShowEQ. I just made it apply cleanly to the latest code and changed the opcode and struct to work with EQLive.

I had the same problem with thing.pl not dumping the databse. It creates the database file (which is always 0 bytes long) and some other log file, but it doesn't remember prices between runs.

Unfortunately I don't know Perl and don't know what the problem is.

[perlmonkey]

So, if I were writing my own version of this, and I wanted to translate the merchant id to a spawn name, how would I do that? I cannot seem to find a publicly accessible SpawnShell, on which to call FindID with my merchant ID from the bazaar search response.

In interface.c, a SpawnShell is instantiated as a private member, but I don't see any API in there for accessing it.

[perlmonkey]

I answered my own question, after exploring the relationship between signals and slots (never done any Qt work before, as I've been working mostly under Gnome).

Here is a patch that works today. Advatages over other approaches include:

Merchant name is given instead of ID
Output is to a log file in standard EQ log format
stdout messages indicate some additional fields
Detection of search strings
By doing the work via a signal/slot this is more in line with the rest of the code

BUGS/TODO

Too much stdout
On-the-fly pointer arithmetic rather than struct casting (due to variable stuct sizes)
Relationship between fields for some search types not yet understood
There seem to be more reasons than just the search UI that these packets are sent. Currently such cases are warned about and otherwise ignored.

Enjoy!

PS: note that this is against the mainline. I have not yet explored 5.0

[tamasine]

excellent - i've never understood the API running the show in seq which is why i ignored it, and patched all the stdout stuff. my hat is off to you :)

now all that's needed is for someone to write a bazaartracker/thing.pl clone in c++ integrated with seq *grin* [preferably someone who knows stats - i calculated mean prices, but time-weighted means would be more helpful as would std.dev and other scary stats numbers]

[perlmonkey]

Originally posted by tamasine
now all that's needed is for someone to write a bazaartracker/thing.pl clone in c++ integrated with seq *grin* [preferably someone who knows stats - i calculated mean prices, but time-weighted means would be more helpful as would std.dev and other scary stats numbers]

The reason that I spit this out in EQ's log format is that I already have all the statistical analysis work done for straight logs. It will take some time to clean up and finalize that code (there is a LOT of it), but I will release it as open source when I'm done.

What I use is a time-weighted percentile. For buying and selling, I buy at the tenth percentile price or below, and sell at the 30th percentile. I also have some reporting and graphing stuff as well as a bunch of CGI-based canned queries. If I could afford the bandwidth and CPU, I would just open up the version that is runnking on my server, but I can't really do that.

Right now everything depends on MySQL 4.1, but you could probably spend less than a couple of days and get it to work with just about anything. I have tried to only abuse MySQL-specific things where they were really needed.

Soon....

[perlmonkey]

Up to this morning, and hopefully still after the patch, this is working like a charm. My code cleanup on the database and statistics side is progressing nicely. I'm going go speed release by breaking this all into 3 parts: the parser and database injector; the CGI and graphing; and the data analysis and reporting.

The first part I should be able to release in just a few weeks, along with a sample database schema with some seed data.

[Reason]

Weeks? /cry

Hehe... I'm almost as excited about this as I am about the new expansion. :p

[Dark]

Is there anywhere i can get this as a complete version of this with all the patches etc included?

[perlmonkey]

Originally posted by Dark
Is there anywhere i can get this as a complete version of this with all the patches etc included?
Dark, there are several "thisses" in question. Do you mean the original bazaar search patch? That is the first patch in this thread, but AFAIK, no longer works.

My version is a few pages back, and is a standalone patch. As with most things seq related, there's a bit of a technical barrier to entry. If you're asking if I will apply the patch for you, no. There is no secret involved; applying patches is not a dark art, and unless I get commit rights to seq, I doubt this will be mainstream any time soon.

Now, if you are talking about the (totally unrelated to seq, other than its ability to read the EQ-log-like output of my aforementioned patch) pricing management system, I will be publishing that as a (set of) stand-alone package(s) sometime in the next few weeks, as real work permits.

That software will probably ship with the patch for seq, though it won't require you to use it (I have been using it on regular EQ logs for months now).

Reason, thanks for the compliment, I hope it lives up to your expectations. It's taking so long because I'm going back over 2700 lines of Perl code and trying to clean it up and make it maintainable ;-)

PS: Expect a new version of my patch to be posted as soon as seq works again. It will be almost the same (modulo any tweeks I've made) but will be based on whatever the "current" code-base is at that point.

[Dark]

Thx for the reply. It was the stand alone version i was interested in. I'll await you release.

[perlmonkey]

Bad news from the front lines... The update from SOE was pretty heavy on the protocol changes. It looks like seq will take a while to fully patch / recover, and THEN can I start the work of fixing this patch. It may also delay work on releasing the database code :(

On the bright side, I finished some documentation last night after I gave up on opcodes.

[Fatal]

Can I ask if you have decided how you are going to do the back end on this?

I'm assuming that once you fetch the pricing data you can pretty much put it where you want, such as a sql database and then manipulate the data in any manner ya want such as a php webpage etc. Or just an app that interacts with a database.

I assume you are approaching it in one of those ways, but wa just curioous how were you doing it.

My apologies if you have stated already and I missed it

[perlmonkey]

Originally posted by Fatal
Can I ask if you have decided how you are going to do the back end on this? You can ask, and you'll even get an answer ;-)

I wrote the back end long before I wrote this patch. It's just a log parser that reads pricing data from EQ format logs and writes to (as you postulated) an SQL database.

There are other parts for doing further analysis, but that's basically it. I just needed the SEQ piece to gather the data faster and with less user overhead.

[perlmonkey]

Well, now that seq is working again, I am working on a new version of my patch. It will be a bit different. Zaphod has talked me into changing the logging a bit, so I'll do that.

Once that is done, I will post the 1.0 version of PDB. PDB is just the log parser and the Perl modules that support it for talking to the database.

Then, I will start work on all of the front-end tools. I have a TON of code there, so it will take some time, but getting PDB working and feeding it data from your logs and from seq should take a little while, so by the time you have anything to analyze in your database, I should have the CGI analysis and graphing tools ready for release.