What I've been up to... (aka ShowEQ 5)

[Zaphod]

Lately I've been working on some major changes to ShowEQ. These changes are very pervasive and aim to both clean up a lot of the code and add new/improved features. When I am done (or at least decide to declare victory) we will have ShowEQ 5.0. I have decided to do this development in a seperate branch within CVS to insulate the regular users from these changes until they stabilize. Once certain milestones are achieved and certain features stabilize I will be releasing beta versions.

Some of the changes related to this development have already been included in (4.3.17, 4.3.17.1, and 4.3.18).

Here are other changes that have already been completed or are nearing completion, but won't be released in 4.3.x:
Added zone safepoint support to ZoneMgr
Some dependency cleanups.
Fix to support people running EQ under winex.
Switch to using QRegExp for filter pattern matching.
Major cleanup/revamp of Filter handling (stage 1), filters are now stored in their own sub-directories, and now uses an XML format.
Created a perl script filter2xml.pl which can be used to convert existing filter conf files to the new XML format.
More major cleanup of packet handling (stage 2). Migrated opcode,implicitlen and dispatch handling to an XML based system (see worldopcodes.xml and zoneopcodes.xml).
Modified logging of decoded world/zone packets to include which packet in the opcode DB they matched, when the entry was updated, and information about the type that is supposed to be in the payload.
Fixed the spawn logger to log the adjusted x/y/z values for zone/new spawns..
Slight improvement in map loading speeds.
Properly support seperation of user data from distributed program data, including support for a .showeq directory or the like
Migrated SEQWindow to be based off of QDockWindow so that all classes derived from it could take advantage of Qt's native dockable window support.


Here is what I currently see as the TODO list for 5.0 (this list will change with items being added and removed):
Packet handling:[list=1]
Add support for an EQPacketSession concept that will support one SEQ process to monitor N EQ clients.
Organized packet capture display with the ability to filter in/out different
packets.
More general packet cleanup.
[/list=1]
Better support for running with SetUID.
Move all non-GUI related stuff out of EQInterface
Move terminal UI to its own class
Move eqstr_{}.txt handling to its own class
Improved filtering system
Map: [list=1]
Support multiple user configurable icons (for spawns, pc's, items, etc...)
Support display of zone safepoint (from zone packet)
Support more map editing capabilities.
Support simple map optimization
Support rescaling the Z for a whole map to allow migration of old/borked maps.
[/list=1]
Move maps, filters, spawnpoints, and logs to their own subdirectories.
New message storage, filtering, and UI.
Support for LoY guild information capture/display.
Redo the base UI so that it is loaded from a QT .ui file to make it easier for the users to reconfigure/add stuff.
Seperate UI from data storage/handling in Combat Window and Experience Window
Create new and improved item db support.
Improved group/guild/raid support.


If people have specific feature requests I am willing to entertain them (or at least be entertained by them). Also, if people wish to volunteer to contribute to this effort it would be greatly appreciated.

Going forward my 4.3.x work will be limited to opcode and structure changes to keep up with EQLive as well as the occasional bug fix.

Thanks and Enjoy,
Zaphod (dohpaZ)

[S_B_R]

Have I mentioned, it's good to have you back Zaphod? :D

It's corny I know, but how about a DPS graph in the Combat window? Hit Damage on the Vertical and Time across the horizontal. Mob DPS could be displayed similarly, or even on the same graph in a different color.... :)

Anyway, Thanks for all your hard work Z.

[guice]

Very nice!

Might I make one suggestion? Migrate the Map parsing to use the same format as SOE's. This will assist in cross compatability with in game maps and SEQ.

Didn't see it in your to do list, so thought I'd suggest it.

[Tor K'tal]

Originally posted by guice
Very nice!

Might I make one suggestion? Migrate the Map parsing to use the same format as SOE's. This will assist in cross compatability with in game maps and SEQ.

Didn't see it in your to do list, so thought I'd suggest it.

ACK... how about just double support based on file extention. I know it isn't really the linux way but it would be an easy way to determine which parsing system to use. The reason I say this is because the SEQ map file is technicly superior in file size and other aspects.

How you might ask? The simple answer is the way the line points are stored. SEQ uses x1,y1,[z1,]x2,y2,[z2,]x3,y3[,z3] while SOE uses x1,y1[z1,],x2,y2,[z2]x2,y2,[z2],x3,y3[,z3]. As you can see that will quickly add up to be a much larger files in the long run. Larger files take more time to parse. Also ingame editing of the map in EQ does some weird stuff where it actually can start putting only x1,y1,[z1,]x2,y2[,z2] on a line then putting the next on a new line so it doesn't actually string them all together, meaning you have a line diffinition character, a color set, a point count too. Currently both file formats support the same way to define a color by using an RGB system while SEQ also has the ability to understand pre defined color names such as green and blue and grey and about 12 others. SEQ RGB is formated #RRGGBB in hex and SOE closes their lines with 0 to 255 R, G, B.


So Zaph, got to say what great work you have done so far and I much appreciate it. Support of both file formats wouldn't be too tough based on file extention, or even prompting the user to select SEQ or SOE format if the extention wasn't a .map or a .txt file when they open it manually. The system itself would try and use a .map file if it existed then try for the .txt and if neither existed, no map would be loaded apon zoning.


I think it is an awesome idea to move the opcodes to an XML style system. I just wish I knew more about XML in order to help you with that. I was wondering if you could put the commonly refered to structure in a similar XML sytle format. I am not 100% clear on what this structure is, my believe is it's actual formating of the EQ packets themselves but I could be wrong. I just know that it occationally changes to.

I have a question on the XML for the opcodes, with this system will you be able to edit the opcode on the fly refresh the parse of the XML file and then start interpruting the next amount of data with the fresh changes? I don't expect it to be retro active against data you have already miss interpruted.

So again, great job and keep up the awesome effort.

~ TK

[Zaphod]

Originally posted by Tor K'tal
ACK... how about just double support based on file extention. I know it isn't really the linux way but it would be an easy way to determine which parsing system to use. The reason I say this is because the SEQ map file is technicly superior in file size and other aspects.

I agree. I find the SOE map format to be wasteful both in terms of disk space and processing time. It looks like the work of a very junior level software engineer.



So Zaph, got to say what great work you have done so far and I much appreciate it.

Glad to hear it.



Support of both file formats wouldn't be too tough based on file extention, or even prompting the user to select SEQ or SOE format if the extention wasn't a .map or a .txt file when they open it manually. The system itself would try and use a .map file if it existed then try for the .txt and if neither existed, no map would be loaded apon zoning.

I've been thinking about doing that. It would be fairly trivial to support reading/writing both formats. It's a lower priority item, but could be accomplished in little time.



I think it is an awesome idea to move the opcodes to an XML style system. I just wish I knew more about XML in order to help you with that. I was wondering if you could put the commonly refered to structure in a similar XML sytle format. I am not 100% clear on what this structure is, my believe is it's actual formating of the EQ packets themselves but I could be wrong. I just know that it occationally changes to.

The structures in everquest.h do correspond to the format of the packets themselves. I have been thinking of moving the structures to an XML based system as well, the main concern I have with doing that is most, if not all, of the techniques you would use to do that would be slower then just mapping it to a C style structure. The biggest advantages we'd get are the ease of programatic editing, possibility of runtime correction, minimizing the need for recompiles, and the possibility of supporting big-endian platforms (ala Solaris Sparc, IRIX, and MacOS X).



I have a question on the XML for the opcodes, with this system will you be able to edit the opcode on the fly refresh the parse of the XML file and then start interpruting the next amount of data with the fresh changes? I don't expect it to be retro active against data you have already miss interpruted.

I might support a reparse of the file, but I'm also thinking about the possibility of providing a UI, possibly as part of a packet analysis UI, that allows you to edit the opcodes within ShowEQ while it was running. At which point the change would take effect immediately and then you'd be able to just save your changes to the XML file. This could make opcode hunting much simpler.



So again, great job and keep up the awesome effort.

~ TK

[uRit1u2CBBA=]

How about this:

It would only affect the first time that an SOE map is read per zone, so the overall overhead vs. using only the current SEQ maps are minimal.

I zone into a new zone. Call it ZZZ since it would be the same anywhere.

I look for file named ZZZ.map. If exists, cool, use it. done.

If it doens't exist, then look for ZZZ.txt. If it exists, read it, then write it as a .map, and delete the .txt (if you want to save space).

Now the .map file exists for the next time I zone into ZZZ.

And of course if it doesn't exist in any form, no map, just as it is today.

[Cryonic]

You can almost do that now, it's just not automated. See the Tools Menu, click it the next time you have SEQ open.

[datadog]

Wow Zaphod.. this is totally awesome...

As far as feature requests. Im thinking of some changes to the buff window.

1. Possibly have separate windows for buffs and detrimenal spells like dots and mezzes?

2. Would it be possible to filter the buffs listed in the window to the currently played character (either target or caster)?

Not being a programmer of any ilk, i have no way to know if this is feasible, but I thought I'd ask. :)

Thanks again for all of this hard work. It is greatly appreciated.

[Velium_hound]

Nice list!

One thing that would be "nice" to have is the a hook for event (packet) listeners. Basically a class that I could add my own instances to which would get copies of the events, decide if it was a event type this listener is interested in, and do something with it.

(Like prices in the bazaar)

The point though is that people could drop in their own experimental toys easily given a standard interface. Then if one of particular merit surfaces, integration is relatively simple.

Personally, I think it would be rather simple to code up a little add-on that might track DPS or "complete heal" trending on a raid. Or alert me when that person hails the PP a second time (or third...).

[jesterbam]

A big fat THANK YOU !!!!!! Amazing skills.

[guice]

I agree. I find the SOE map format to be wasteful both in terms of disk space and processing time. It looks like the work of a very junior level software engineer.

I can agree to that. The idea here is that the SOE Maps is in a format we don't control while the SEQ format is under the SEQ dev's control. It would only make sense to convert what the devs have control over to what they don't have control over, if that makes sense.

Even though SOE's is very poorly done, there isn't anything the SEQ devs, that I know of, can do to fix it. So, just go with the flow.

[monklett]

More robust configuration in logging of spawns, times, locs, etc.

This is the sort of data that is necessary to sort out many quests, etc. (I still haven't quite figured out the SRo AC spawn, although I have killed him over a dozen times).

Even a simple switch to just dump all this to disk, without packet information, would be nice.

[Zaphod]

Originally posted by monklett
More robust configuration in logging of spawns, times, locs, etc.

This is the sort of data that is necessary to sort out many quests, etc. (I still haven't quite figured out the SRo AC spawn, although I have killed him over a dozen times).

Even a simple switch to just dump all this to disk, without packet information, would be nice.

Ummm, most of what you just mentioned can be gotten out of the spawnlog (which doesn't contain packet information). So, what are you requesting that is different then the spawn log?

Enjoy,
Zaphod (dohpaZ)

[monklett]

Originally posted by Zaphod what are you requesting that is different then the spawn log?

I apologize; that post sucked. I shouldn't have flashed it out before bed. What I actually had in mind was log file support for Xylobot and other 3rd party apps.

You see, Xylobot can only accept input from a log file or by reading the colors of specific pixels on the screen (imagine a black screen with only your health/mana bar and chat window). With some programming, like looping a /loc key, you can do rudimentary things like move to a certain location or face a general direction. However, you cannot do things like chase a fleeing mob, or determine the distance between 2 mobs, or even (reliably) face a specific direction.

Of course, Macroquest can do all these things and more. However, it has its own issues in that: (1) its far more detectable, because of the way it reads and writes eqclient information, (2) its scripting language is much less robust (although Xylobot's isn't great either), and (3) it has been available in a functional state far less often than Xylobot over the last couple of years.

However, if ShowEQ could be coaxed into writing only a couple of facts to a disk file every second or so, Xylobot libraries could be easily written to perform nearly all of Macroquest's advanced features, without its issues.

These facts are:

1) your current loc and the direction you are facing in degrees. (I believe EQ's precision is only 2 degrees--at least that true for the client display--I am not sure how it is stored internally).

2) the loc and direction of your current target.

3) (optionally) the loc and direction of any mobs within configurable distance X of your loc.

An update would be made when values change, up to once per second.

When I looked into pulling this information out of the spawn file a few weeks back (you may remember my post about the changes in logger.cpp), it became evident that the best way to do this would be to use a perl or python script to parse the spawn log, strip out the relevant information, and then write it to another file. A better way would be through a config option within ShowEQ to output this information separately, which is what I was looking at doing when I made that post.

(Actually, the right way would be to connect directly to Xylobot via TCP, etc., but that has its own issues. Using the custom log file approach is adequate, well supported within Xylobot, and would allow this information to also be used from any other 3rd party app without any additional tweaks or support issues from the ShowEQ team).

Anyway, I have written some fairly complex Xylobot scripts (including one that will grind AAs without human intervention), and am quite familiar with its quirks and limitations. If this sounds like a feature that you would be interested in, I can provide much more detail on what to output, etc.

[Reason]

Bazaar tracking. /nod/nod =)

Min/max/ave/trends stuff like that. plz.

[humble1]

something down near the bottom bar..or actually on it (where run speed, EQ time and date and zone location), that has a timer for LDoN missions, the 30mins before entering, the 1.5 hours inside, and the 30 mins after...this info is sent to client no? i posted this in feature requests as well. I just think this would be a nifty lil feature to have.

Also: actually i feel this is more important....i use the spell timer frequently, i'd like to see me be able to display only the spells i cast and time left, also be able to clear it (individually and completely) like the spawn window.

Thanks for your time and effort!

[Cryonic]

You can clear spells from the spell list by double-clicking the spell.

[Dedpoet]

You can clear spells from the spell list by double-clicking the spell.

This is true, but I think I'd still like to see the ability to exclude certain spells from going in the box, as discussed in another thread (http://seq.sourceforge.net/forums/showthread.php?s=&threadid=4270). Clicking off 25 instances of Celestial Tranquility, for example, after a fight is not enjoyable.

[Circles]

a few recommendations:

search feature in the logging (channel message) window

the abiltiy to right click spawn points, mobs, etc and delete them

option that when a map is found that EQ will not expand beyond its boundries to show spawns (instead just show the spawn at the edge of the screen with a roow indicating its direction off screeen)

instead of a zoom multiplyer, allow zoom by size (ie 200 400 eq units)

[Rufus]

Originally posted by Dedpoet
This is true, but I think I'd still like to see the ability to exclude certain spells from going in the box, as discussed in another thread (http://seq.sourceforge.net/forums/showthread.php?s=&threadid=4270). Clicking off 25 instances of Celestial Tranquility, for example, after a fight is not enjoyable.
While we're on the subject of the spell list, I've got two pet peeves that should be easy to fix (I've been meaning to do it for awhile but I just haven't had the time to dig through the code):

* Removal of a spell from the list when it gets refreshed (ie: If I've already got Virtue and somebody refreshes, the original one should be removed immediately instead of keeping both copies with seperate timers)

* Removal of all spells from the list when you die (without checking I can't recall if we've got the opcode for player deaths but we could always just clear the spell list when we start zoning so that it can be repopulated from the buff list received with the new zone info).

Rufus

[Cryonic]

Originally posted by Rufus
While we're on the subject of the spell list, I've got two pet peeves that should be easy to fix (I've been meaning to do it for awhile but I just haven't had the time to dig through the code):

* Removal of a spell from the list when it gets refreshed (ie: If I've already got Virtue and somebody refreshes, the original one should be removed immediately instead of keeping both copies with seperate timers)


The only buffs that SEQ seems to have issues with (for me anyways) are the self only buffs. Cast SOW on myself or others and it refreshes the timer for that person.



* Removal of all spells from the list when you die (without checking I can't recall if we've got the opcode for player deaths but we could always just clear the spell list when we start zoning so that it can be repopulated from the buff list received with the new zone info).

Rufus

I don't like this idea. If it could remove the spell on death, but not on me zoning as that would remove timers that I might be watching (e.g. buffed up outside a zone, like PAW, and then zoned in. Now I only know when my buffs will be gone).

[Dedpoet]

I don't like this idea. If it could remove the spell on death, but not on me zoning as that would remove timers that I might be watching (e.g. buffed up outside a zone, like PAW, and then zoned in. Now I only know when my buffs will be gone).

Is this true? I thought the buff info was sent every time you zone, just that it's not cleared. You can test this by restarting seq and zoning, or starting it up fresh and logging in. The problem is that they aren't cleared when you die/zone, so the timers stay up even though the buff isn't there anymore. If you cleared it when you zoned, it would just re-populate when it got the new data, and if you had died and had no buffs, it would be empty. Right?

[Zaphod]

Originally posted by monklett
I apologize; that post sucked. I shouldn't have flashed it out before bed. What I actually had in mind was log file support for Xylobot and other 3rd party apps.
I won't be adding specific support for macroing programs. If you wish to create macros that need specific knowledge about the internal state of EQ, then I suggest you use MacroQuest. In recent months MacroQuest appears to have been quite stable (since the return of Plazmic). It is only slightly more detectable then Xylobot, anything running on the same physical or virutal machine as eqgame.exe is detectable. Don't delude yourself into thinking otherwise.


Originally posted by humble1
something down near the bottom bar..or actually on it (where run speed, EQ time and date and zone location), that has a timer for LDoN missions, the 30mins before entering, the 1.5 hours inside, and the 30 mins after...this info is sent to client no? i posted this in feature requests as well. I just think this would be a nifty lil feature to have.

Also: actually i feel this is more important....i use the spell timer frequently, i'd like to see me be able to display only the spells i cast and time left, also be able to clear it (individually and completely) like the spawn window.

Thanks for your time and effort!
Better support for LDoN missions would be a good thing, and if I or someone else can find the appropriate opcodes and structures to support them I'll see about adding that support. I'm not planning on going out of my way to find those bits of data right now though.

You currently clear individual items from the spell window, just double click on the item you wish to remove. As far as providing a manual way to clear it completely, I can see adding that.


Originally posted by Dedpoet
This is true, but I think I'd still like to see the ability to exclude certain spells from going in the box, as discussed in another thread (http://seq.sourceforge.net/forums/showthread.php?s=&threadid=4270). Clicking off 25 instances of Celestial Tranquility, for example, after a fight is not enjoyable.
Adding filtering for spells sounds like a distinct possibility. It would be a lower priority item on my rather large todo list though. It would definitely help if more people were submitting code or offering assistance rather then just suggesting additional work.


Originally posted by Circles
a few recommendations:

search feature in the logging (channel message) window

the abiltiy to right click spawn points, mobs, etc and delete them

option that when a map is found that EQ will not expand beyond its boundries to show spawns (instead just show the spawn at the edge of the screen with a roow indicating its direction off screeen)

instead of a zoom multiplyer, allow zoom by size (ie 200 400 eq units)
A search feature for the message window is already in the works (The message handling and message window are in the process of being completely redone).

You can already right click on spawn points and delete them. I'm not sure what the utility of right-clicking and deleting individual mobs would be. If you are just trying to hide that particular mob, just add it to the Filtered filter.

I'm not sure what the point of not expanding the maps would be? If it is only prevent the odd map compression/expansion/blow up, whatever you choose to call it where a spawns location is misinterpreted, then I'd say the best way to solve that is to fix the misinterpretations. There are many maps that are incomplete and don't show the entirety of there zones, so at best I feel that such a feature would be of limited utility.

As far as zooming by a specific size, that would require changing some core logic in how zoom handling is done throughout that map code. I'm not about to tackle that problem at this juncture.


Originally posted by Rufus
While we're on the subject of the spell list, I've got two pet peeves that should be easy to fix (I've been meaning to do it for awhile but I just haven't had the time to dig through the code):

* Removal of a spell from the list when it gets refreshed (ie: If I've already got Virtue and somebody refreshes, the original one should be removed immediately instead of keeping both copies with seperate timers)

* Removal of all spells from the list when you die (without checking I can't recall if we've got the opcode for player deaths but we could always just clear the spell list when we start zoning so that it can be repopulated from the buff list received with the new zone info).

Rufus
The first part should work, but occasionally doesn't. That' just a bug that needs to be fixed, feel free to submit a patch. I may or may not find the cause during the other work I'm doing.

The second part mainly applies to spells cast upon your toon, but you shouldn't have it globally clear spells that you cast upon someone else when you zone. That would make the spell list next to useless for buffing caster classes.


Originally posted by Dedpoet
Is this true? I thought the buff info was sent every time you zone, just that it's not cleared. You can test this by restarting seq and zoning, or starting it up fresh and logging in. The problem is that they aren't cleared when you die/zone, so the timers stay up even though the buff isn't there anymore. If you cleared it when you zoned, it would just re-populate when it got the new data, and if you had died and had no buffs, it would be empty. Right?

The point they are trying to make is that they are trying to track the buffs they cast on others. That info is only sent when the buffs are initially cast upon the other person. The only buffs received when you zone are those cast on yourself.

Enjoy,
Zaphod (dohpaZ)

[Rufus]

Originally posted by Zaphod
The first part should work, but occasionally doesn't. That' just a bug that needs to be fixed, feel free to submit a patch. I may or may not find the cause during the other work I'm doing.

The second part mainly applies to spells cast upon your toon, but you shouldn't have it globally clear spells that you cast upon someone else when you zone. That would make the spell list next to useless for buffing caster classes.
D'oh! Would you believe that I didn't even realize that the spell list also tracked spells that you cast on others? I guess there's a reason they don't let us warriors cast spells. ;) So yes, only your own personal buffs should be cleared from the list on zoning.

I'll see what I can come up with for a patch for these two items.

Rufus

[lildr00d]

I'm not sure what the point of not expanding the maps would be? If it is only prevent the odd map compression/expansion/blow up, whatever you choose to call it where a spawns location is misinterpreted, then I'd say the best way to solve that is to fix the misinterpretations. There are many maps that are incomplete and don't show the entirety of there zones, so at best I feel that such a feature would be of limited utility.


Im not sure if my idea would be easier then fixing the problem or not. What about the ability the Drag a box over the map area to set a ''filter'' of sorts. I.E. Dont decode any data outside of box area. You could drag it over the map area or if you are in a larger zone on a slow PC you could just drag the ''filter box'' over the area you are concerned with. Not sure if this is a good idea or not. Just a idea that I thought of.


Adding filtering for spells sounds like a distinct possibility. It would be a lower priority item on my rather large todo list though. It would definitely help if more people were submitting code or offering assistance rather then just suggesting additional work.

I truely wish I was a wizard with the code as much as you are Zaphod then I could actually try my idea's out and make more of a difference here.

[perlmonkey]

Originally posted by Reason
Bazaar tracking. /nod/nod =)

Min/max/ave/trends stuff like that. plz.

Oh please, who hasn't written that themselves already... ;-)

What I do want to do is start dumping bazaar search results to a file, so that I can add that to my log file analysis, but that's about all I'd need to add to my current tracking. I'll be releasing my bazaar management stuff as open source at some point (basically awaiting code cleanup now), which includes trend analysis, graphing, bzr_*.ini file generation, graphing, etc. My database hase somewhere around 4000 validated prices now (validated meaning I've seen enough data-points that I feel confident in the pricing). I'll announce it here when it reaches non-alpha status.

[Reason]

Originally posted by perlmonkey
Oh please, who hasn't written that themselves already... ;-)

What I do want to do is start dumping bazaar search results to a file, so that I can add that to my log file analysis, but that's about all I'd need to add to my current tracking. I'll be releasing my bazaar management stuff as open source at some point (basically awaiting code cleanup now), which includes trend analysis, graphing, bzr_*.ini file generation, graphing, etc. My database hase somewhere around 4000 validated prices now (validated meaning I've seen enough data-points that I feel confident in the pricing). I'll announce it here when it reaches non-alpha status.

Sweet! I can't wait!

[R0x0r_Mc0wnage]

Dophaz: As a note, SEQ v5 does not have the fedora issues I reported in SEQ v4. Nots ure if it's the new QT or what, but it works. Compiled with -no-xft :)

[Nurseling]

My first suggestion is to have the ability to move anything you do no want docked on the main window (ie. secondary maps, message box, exp window, combat window) to another workspace.


This is a little further out there but may be easier to accomplish. I would like to be able to place an audible alert on some spells before they expire.


I hope these suggestions entertain you and thank you for all your hard work.

[someshaman]

I currently play a beastlord, I often pull and I often mistake my pet as a mob :D It would be nice to be able to set certain NPC to different colors, like pets, shopkeeper, bankers.

[vylesilencer]

Aye, easy differentiation of pet and mob would be very handy indeed.

[uRit1u2CBBA=]

maybe a square like PCs look like, but a different color border?

[dogmeat]

Thanks for all the work, Zaphod, and everyone. So then, would you recommend I take the plunge and upgrade to RH9 from RH7.1, to use the new 5.0 beta and subsequent versions?