A new Lucy

[ksmith]

__________________________________________________
Edit: update. original post below.

I'm looking for more people to test the item collector. Installation Instructions are here (http://eqitems.13th-floor.org/install.php).
__________________________________________________

Since it doesn't look like Kerasota is going to be updating lucy to work with LDoN or releasing the source code, I've started work on a new item database.

Currently I just have showeq dumping the data from a couple of opcodes to a file and then I run that through a couple of perl scripts to generate some happy html.

The most important thing I need now, is to get a proper item collector program written. I've been looking at Java and Jpcap since I don't have visual studio and it would allow it to run on linux and windows.

Here's a sample of what I've got:



Royal Attendant Breastplate of Sand

MAGIC ITEM LORE ITEM NO DROP
Slot: CHEST
AC: 69
STA: +13 CHA: +11 WIS: +10 INT: +10 HP: +100 MANA: +100
SV FIRE: +12 SV MAGIC: +12 SV POISON: +12
Reccommended level of 65.
WT: 3.5 Size: MEDIUM
Class: WAR CLR PAL SHD BRD
Race: ALL
Slot 1, type 2: empty
Slot 2, type 2: empty
Slot 3, type 3: empty


itemclass 0
name Royal Attendant Breastplate of Sand
lore *Royal Attendant Breastplate of Sand
idfile 0
id 44145
weight 35
norent 255
nodrop 0
size 2
slots 131072
price 0
icon 624
012 -1
013 0
UNK014 0
tradeskills 0
fr 12
dr 0
pr 12
mr 12
cr 0
str 0
sta 13
agi 0
dex 0
cha 11
int 10
wis 10
hp 100
mana 100
ac 69
UNK031 0
skillmodvalue 0
skillmodtype 0
banedmgrace 0
banedmgamt 0
banedmgbody 0
magic 1
casttime_ 0
level 0
reqlevel 0
bardtype 0
bardvalue 0
light 0
delay 0
reclevel 65
UNK046 0
elemdmgtype 0
elemdmgamt 0
effecttype 0
range 0
damage 0
color 13107350
classes 151
races 32767
UNK055 0
spellid -1
maxcharges 0
itemtype 10
material 5
060 1.000000
061 0
casttime 0
063 0
064 0
focusid 0
combateffects 0
067 0
068 0
strikethrough 0
070 0
071 0
072 0
073 0
074 0
075 0
factionmod1 0
factionmod2 0
factionmod3 0
factionmod4 0
factionamt1 0
factionamt2 0
factionamt3 0
factionamt4 0
charmfile
augtype 0
augslot1type 2
augslot2type 2
augslot3type 3
augslot4type 0
UNK090 0
ldontheme 5
ldonprice 1492
ldonsold 1
bagtype 0
bagslots 0
bagsize 0
bagwr 0
book 0
099 0
filename
101 0
102 0

[junk]

Very very cool, ksmith. The ONLY thing I ask (if you're so inclined) is to throw some kind of database abstraction layer in there so we can directly output data to a database of our own (I'm thinking MySQL for myself, but whatever anyone might want to write a plug-in for). Either that or at least document the table structure for whatever output format you choose =).

[Zaphod]

I'm currently in the process of updating the item capabilities of ShowEQ to deal with the current EQ packet format. I've currently got my own notes, some from UnGod, and have been looking at what the EQEmu guys have figured out. But if you would care to share your own notes/data/code for dealing with the new items it would be greatly appreciated because there are still fields that none of us have figured out yet. So, please share your insight.

Enjoy,
Zaphod (dohpaZ)

P.S. I'm planning on a CVS commit with item support in the Tuesday/Wednesday timeframe.

[ksmith]

Originally posted by junk
Very very cool, ksmith. The ONLY thing I ask (if you're so inclined) is to throw some kind of database abstraction layer in there so we can directly output data to a database of our own (I'm thinking MySQL for myself, but whatever anyone might want to write a plug-in for). Either that or at least document the table structure for whatever output format you choose =).

I intend to do something like that. One of the things that I didn't like about lucy was all the hoops you had to go through to get lucy item data into a guild loot/dkp database.

[ksmith]

Originally posted by Zaphod
I'm currently in the process of updating the item capabilities of ShowEQ to deal with the current EQ packet format. I've currently got my own notes, some from UnGod, and have been looking at what the EQEmu guys have figured out. But if you would care to share your own notes/data/code for dealing with the new items it would be greatly appreciated because there are still fields that none of us have figured out yet. So, please share your insight.


I'm watching the ItemCode and ItemInfoCode opcodes. The data is '|' delimited strings rather than anything resembling a C struct.

Here's the important parts of what I put in packet.cpp:

// print out contents - data is '|' delimited strings
FILE *itmFile;

// on bazaar queries data[6] == 0, filter them out
if (data[6] && (itmFile = fopen("./items.raw", "a")))
{
for (int i = 6; i < (int)len - 1; i++)
fprintf(itmFile, "%c", (char)data[ i ]);
fprintf(itmFile, "\n");
fclose(itmFile)
}

Everything before data[6] doesn't seem to have anything to do with the item data, and data[len - 1] is just a NULL. That leaves us with


1|0|4|0|0|"0|Earring of the Solstice|*Earring of the Solstice|IT63|28771|
1|255|0|0|18|1000|1072|-1|-1|0|0|5|0|0|5|5|0|5|0|0|
0|8|8|55|55|5|0|0|0|0|0|0|0|0|44|0|0|0|0|0|51|0|0|
0|2|0|0|0|32767|32767|0|1298|-1|29|0|1.000000|0|
0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0||0|1|
0|0|0|0|0|0|0|0|0|0|0|0|0||0|0"||||||||||

Well, that refused to properly line up at all, but it's all one big string, no spaces or newlines, just 102 '|' delimited fields.

Our work is not done yet though. Containers and items with augments add in some extra fun stuff so that all the information about the container and its contents are sent in one packet. Container packets have the same opcode, but the data looks like this:


junk "item" "junk \"item\" junk" "junk \"item\" junk" junk

To get the item data out of that, I break up the whole thing like this:


@fields = split(/\|/, $data);

for $i (0..$#fields) {
$value = $fields[$i];

if ($value =~ /^\"./ && $fields[$i+5] !~ /^\\\"/) {
print_item($i);
}
if ($value =~ /^\\\"/) {
print_item($i);
}
print "\n";
}

The function 'print_item($i)' just prints from $data[$i] to $data[$i + 102] into a file.

As for the field names, they're in exactly the same order that is listed in my first post in this thread.

PM me if you want to know more Zaphod.

[S_B_R]

I think this will be a great utility keep up the good works guys :) I know it's a bit early fora feature request but I'll through it out anyway. Would it be possible to add location info to the DB?

For example everytime an item is looted from an NPC corpse, awarded from a Quest, or foraged, at that point the zone, /loc and NPC name would be included with the rest of the item data.

As an added bonus, A central DB could then could keep track of how often it drops a given item in each zone.

[AlphaBeta]

I am with S_B_R on this one. Adding a zone, mob name would be great. :D

[ksmith]

I'm not going to even think about anything beyond the basic functionality until I get it working.

[Rufus]

I know you're not taking requests right now but I'm old and if I don't mention it now, I'll forget about it by the time you are. ;)

In addition to the zone/loc/mob info, any chance of parsing out the data when you sell an item to a vendor and adding this to the DB as well?

Rufus

[junk]

http://eqemulator.net/items.php

Go figure.

[LethalEncounter]

Originally posted by junk
http://eqemulator.net/items.php

Go figure.


Hey Whats wrong with our item db? I added as many features as I could think of, and I think that it works pretty well. Coupled with the fact that the data is freely available to anyone I feel this could be the best item db on the net in a few weeks.

[ksmith]

Well, if that had been up before I started working on this, I probably would have said "oh well, guess I'll just wait for their itemcollector."

Now that I've got most of the showeq network code re-written in java, I'm going to keep working on my item collector.

[LethalEncounter]

Ahh, well honestly I was working on this before I even saw this thread.

[junk]

Hehe, I didn't mean anything derogatory about it, LethalEncounter, I just meant "go figure" in the sense that someone finally starts doing this with SEQ and it magically appears elsewhere. :) I think the new item DB and the new packet collector are great ideas, and I'm glad that they were released.

[who_me_use_seq]

Chalk it up to "Great minds think alike" or maybe synchronicity.

Either way, my thanks to the people from both camps who have the time, patience, intelligence and attention span to do this work.

[fester]

I had worked a bit on a program before LDON. With augmentations already in the item data format, I didn't believe it was likely they would alter the data structure of the items when LDON released. I was wrong.

The good news is the new data format is very similar to the spell format used for the last few years. It is much easier to parse items with this format. I suspect they moved to this format to ease their troubles with debugging items when making sweeping changes. I doubt they will ever go back to the C structure format again.

[ksmith]

Edit: scrapped the status breakdown.

As I start work on some of the more complex steps, I'll break down the progress even further.

The web and database work will be going along in parallel with the item collector, but since it's mostly trivial, it doesn't get its own progress list.

During each round of testing, I will take feature requests for both the item collector and the website.

As a closing note, this will require at least jre1.4.2.

[domesticbeer]

Have you considered using SOAP and web services to make DB updates??

[ksmith]

Originally posted by domesticbeer
Have you considered using SOAP and web services to make DB updates??

Currently there is nothing I have planned where SOAP would have a significant (if any) advantage over XMLRPC. XMLRPC being the simpler implementation of the two wins.

XMLRPC is sufficient to allow the client to send new items to the database server. If I decide at a later point in time that transitioning to a SOAP implementation would be beneficial, the code is sufficiently modular to allow that.

If you really want/need the item collector to be able to submit items to your guild's database via SOAP and not XMLRPC, PM me and I'll give you an API to write for so you can write the support classes to send the item data via SOAP.

[ksmith]

My item collector is ready for the first round of testing. Send me a PM if you're interested.

http://eqitems.13th-floor.org/

[fester]

I think I am being picky, but could you not use the same colours as lucy? I am not fond of having to select the text to be able to clearly read it. A white background with blue/black text is very easy for the eyes to read. It seems there are a lot of websites with black backgrounds and light (not bold) text which for me is very hard to read. Is it just me?

Also, one the problems with an open client is fraud. I was planning on attacking this problem by requiring the item to be submitted by multiple people from different ISP's (you can use ARIN's database to match the multiple IP blocks an ISP owns to the same ISP.) Say maybe require 5 copies of the item from 5 different ISPs? That should insure it is a real item while only slightly delaying breaking new items.

[uRit1u2CBBA=]

He does say somewhere that new items will be manually inspected before they are added to the search database on the web site.

I have collected a whole bunch of items -- are they automatically sent to you? Do I have to e-mail them to you?

That part was unclear.

Thanks.

[ksmith]

I personally inspect all items submitted. I also have the option of approving all items in the queue from a specific IP address.

Once the collector sees an item it hasn't encountered before, it immediately sends it to the incoming queue on my server. Such are the joys of multi-threaded programs.


EDIT:
I am using exactly the same colors as lucy, in fact, I'm actually using lucy's css file. Expect the colors and/or layout to change when I am done testing and can spend time to work on the aesthetics of the website.

[ksmith]

I am looking for more people to test/use my item collector (http://eqitems.13th-floor.org). Installation instructions are here (http://eqitems.13th-floor.org/install.php).