Maps with invalid point counts on the line.

[Tor K'tal]

Was wondering if someone could put together a quick and dirty perl script that that corrects the point count in the .map files for the GoDs zones and the few LDoN zones that are broken.

I get, M type line has more points than specified or L type line has more points than specificed frequently.

So my thought was parse the line and check for an M or an L and then start counting points accordingly, if their are more points fix it and if their are less possibly fix it but for sure throw up a message so someone could chose to look at it and see if it should be removed from the file or corrected manually.

Then if we could get the application run against the CVS for the next time folks download the maps, that would be really nice too.

I would do it myself but I don't know perl. If it is too much, point me to a good tutorial on beginner pearl.

-- TK

[Cryonic]

First of all, the language is perl, not pearl, heheh. Second, took me less than 15 minutes to write this and I barely know perl or programming:

<--- Removed ------>

check:

http://kyle.13th-floor.org/eq/maps/sanemap.txt

[Tor K'tal]

Originally posted by Cryonic
First of all, the language is perl, not pearl, heheh. Second, took me less than 15 minutes to write this and I barely know perl or programming: ...


First, you are correct it is perl and not pearl. I did get it right a couple times. Still no excuse for being inaccurate, I appologize.

Second, thank you very much.

Third, doesn't quite fix the issues I'm having as I was hoping it would. But that may be because I wrote the request wrong. In the Qinimi.map, all the M lines actually have a 0 as their point count. I was looking for a reasonably automated way to fix that (I don't relish the idea of manually editing 2000 lines to change one little number to a 2 from a 0), based on the actual number of points plotted on the line. I see that it says how many points were ploted and how many it thought it should have with this peice of info put out to the screen "Invaldi M line at qinimi.map:1836 (2/0)" but it only writes it to screen and not correting the point count number in the file, which is what I was actually hoping for (the write back to the file with the correct number).

SoapBox: Why do so many maps created in EQ only have 1 segment of the line per actualy file line? Does that seem extremely wasteful to anyone else? I realize the logic of automaticly fixing that might be signifcantly more complicate than I'm prepared to ask someone else to do. I vaguely remember at one point someone attempting it though, so I will have to start digging through old map posts.


Thank you again Cryonic for doing as much as you have so far.

The invalid header message was very useful, good catch. It ment I had to manually edit the file to a reasonable number of paramaters to get the script to run against it. Is there an easy way to find the width and hieght and write those into the header as well? I see it as having to track the extreme posative X, Y and extreme negative X, Y then doing a tiny bit of math once it got to the end of the file and jumping back to the top of the file and writing the header line again. All this would mean you would have to look at every perameter on the line, thus slowing down the process significanlty.

Having seen your script I'm becoming more intreseted in seeing a desent beginner's tutorial on the perl language (could be read as, I want to learn more). Thank you again, and a gental shove in the right direction would be much appreciated.

OT: it just dawned me, that a guy's got to make a buck some how. So, if any of you guys know of a decent tutorial for beginners on cultivating pearls be sure to let me know. I hear there is good money in the jewelery business. :)

~ TK

[Cryonic]

The script in the link isn't mine. I originally posted my code here, but Ksmith pointed me to something he had already written some time ago, so I removed my code and added the pointer to his site.


Originally posted by Tor K'tal
SoapBox: Why do so many maps created in EQ only have 1 segment of the line per actualy file line? Does that seem extremely wasteful to anyone else? I realize the logic of automaticly fixing that might be signifcantly more complicate than I'm prepared to ask someone else to do. I vaguely remember at one point someone attempting it though, so I will have to start digging through old map posts.

Simple, because SOE created a map format that is just one pair of points per line. There were scripts posted to the forums to convert from SOE's format to SEQ's and try to reduce the number of lines.

As for making the scripts modify the files, that you might want to take up as a way to learn perl.

The books I have for learning perl (that I just paw through every so often) are Learning Perl from O'Reilly press and Teach Yourself Perl in 21 Days from SAMS Publishing. I also have the Perl Cookbook for some ready made scripts.

[ksmith]

Around line 38 or so:

} elsif (($#line - 3) / 3 != $line[3]) {
print "Invalid M line at $map:$. (".
(($#line - 3) / 3) ."/". $line[3] .")\n";
}


That's where you'd want to make changes if you wanted it to fix the point count. $line[3] is the point count, and (($#line - 3) / 3) is the number of points in the M line.

As for the header, you can just set the max x and max y to zero. Last I checked, showeq doesn't actually do anything with that information.

[Tor K'tal]

I'm trudging away at this and need some feedback before I post a full script to do it here.

Lines with Null's on them, should I attempt to fix the null some how or should I leave the line as is?

Lines with only 1 point, leave them in or strip them out?

Invalid headers, thus forcing an abort, should I force a format to name,name,0,0 ?

Generic invalid lines of other sorts... should they be left in as well or taken out? I have looked at some of the point lines marked as invalid by this script and can't see why they are bad.

Last, how would I go about stripping ^M off the end of the line?

A little note on the trudging comment. I think I am pretty proficent in PHP and I cut my teath in coding on C, so perl isn't that far from either, but it seems to be far enough (taking me longer than I thought it would). For anyone else interested http://www.perl.org is pretty good, but not as good http://www.php.net if you want to learn on your own that is a desent place to start.

~ TK

[ksmith]

Originally posted by Tor K'tal
I'm trudging away at this and need some feedback before I post a full script to do it here.

Lines with Null's on them, should I attempt to fix the null some how or should I leave the line as is?

If the null is in the line name field, replace it with "line". If the null is in the color field, replace it with "gray" or something sane.


Lines with only 1 point, leave them in or strip them out?

Strip them out.


Invalid headers, thus forcing an abort, should I force a format to name,name,0,0 ?

That's what I would do.


Generic invalid lines of other sorts... should they be left in as well or taken out? I have looked at some of the point lines marked as invalid by this script and can't see why they are bad.

Only M, L, P, and H lines matter.


Last, how would I go about stripping ^M off the end of the line?

s/\r$//;


A little note on the trudging comment. I think I am pretty proficent in PHP and I cut my teath in coding on C, so perl isn't that far from either, but it seems to be far enough (taking me longer than I thought it would). For anyone else interested http://www.perl.org is pretty good, but not as good http://www.php.net if you want to learn on your own that is a desent place to start.

man perlfunc

All the manpages that come with perl are very well written.

[Cryonic]

another way to strip off the ^M characters is to use dos2unix on the files

e.g.

dos2unix *.map

[Tor K'tal]

Well, I think I have it done. I haven't been able to find any files it mangaled or broke but that doesn't mean it is 100% safe. Because frankly there are too many files for me to open them all and make a visual inspection of some kind. So I would suggest backing up all your map files before running this against it. This also ONLY works against SEQ format maps. Do not attempt to use it against SOE format files.

Because this is severly under-commented, I appologizes. (infact, I don't think there is a single comment in it). I recommand running it in this fassion ./sanemap.prl > sanemap.log, so that the output can be viewed later, if you care to see it. Also I do not beleive this is elligant or the most efficent possible way to do this, but it gets the job done.

A note on run time, if you aren't dumping to a log file you may think it is hung at Hateplaneb.map but that is because the files is just so huge, please calmly wait. Total run time on my system was roughtly 3 minutes (real = 2m57.472s).

Note on correcting ^M's as line feeds. If the files is otherwise perfect they will not be removed, as the applications will ellect to not write a new file. Maybe Cryonic or someone could enliten me as to some logic to check if the substituion occured and thus set it to write the file (line 63).

While the PHP color coding doesn't quite match perls, it should be close enough if anyone wants to make a visual inspection of the code.


#!/usr/bin/perl -w

opendir(MAPDIR, "./");
my @dir = readdir(MAPDIR);
closedir(MAPDIR);


foreach my $map (@dir) {

next unless -f $map;
next unless $map =~ /\.map$/;

my @newmapfile;

local $mapname = $map;

if(check_sanity($mapname) eq 1) {
write_mapfile($mapname);
}
else {
print "Did not need to write new map file for $mapname\n\n";
}

clear_array();
}


sub check_sanity {
local $map = shift @_;
my @line;
local $linecount = 0;
local $numberofemptylines = 0;
local $numberofMlinesremoved = 0;
local $numberofLlinesremoved = 0;
local $numberofPlinesremoved = 0;
local $numberofNullsfixed = 0;
local $numberofPointCountsfixed = 0;
local $write = 0;

unless (open(MAP, $map) eq 1) {
print "failed to open - $map\n";
return 0;
} else {
print "Opening $map \n";
}


@line = split(",", <MAP>);
$line[$#line] =~ s/\r$//;
if ($#line != 3 && $#line != 7) {
print "$map: invalid header, Forcing a header format of name,name,0,0.\n";
$newmapfile[$linecount] = sprintf("%s,%s,0,0\n", $line[0],$line[1]);
$write = 1
} else {
$newmapfile[$linecount] = join(",", @line);
}

while (<MAP>) {
$linecount++;

$_ =~ s/\r$//;

@line = split(",", $_);
if($#line == 0) {
$numberofemptylines++;
$linecount = $linecount - 1;
$write = 1;
}

if ($_ =~ /\(null\)/) {
print "(null) found at $map:$. - ";
if ($line[1] eq "(null)") {
if (($line[0] eq "M") or ($line[0] eq "L")) {
$line[1] = "line";
print "set the line name to 'line'.\n";
$newmapfile[$linecount] = join(",". @line);
$numberofNullsfixed++;
$write = 1;
} elsif ($line[0] eq "P") {
$line[1] = "Point";
print "set the point name to 'point', consider removing this point or giving it a valid name.\n";
$newmapfile[$linecount] = join(",". @line);
$numberofNullsfixed++;
$write = 1;
}
} elsif ($line[2] eq "(null)") {
$line[2] = "gray";
print "set the color to 'gray'.\n";
$newmapfile[$linecount] = join(",". @line);
$write = 1;
} else {
print "was not able to automaticly fix the null value.\n";
}
}

if ($line[0] eq "M") {
if (($#line - 3) % 3 != 0) {
print "Invalid M line at $map:$.\n";
$newmapfile[$linecount] = join(",", @line);
} elsif (($#line - 3) / 3 != $line[3]) {
print "Invalid M line at $map:$. (". (($#line - 3) / 3) ."/". $line[3] ."), correcting point count\n";
$line[3] = ($#line - 3) / 3;
$newmapfile[$linecount] = join(",", @line) ;
$numberofPointCountsfixed++;
$write = 1;
}
if ($line[3] == 1) {
print "Invalid M line at $map:$. only one point, line removed\n";
$numberofMlinesremoved++;
$linecount--;
$write = 1;
} elsif ($line[3] == 0) {
print "Invalid M line at $map:$. no points, line removed\n";
$numberofMlinesremoved++;
$linecount--;
$write = 1;
} else {
$newmapfile[$linecount] = join(",", @line) ;
}
} elsif ($line[0] eq "L") {
if (($#line - 3) % 2 != 0) {
print "Invalid L line at $map:$.\n";
$newmapfile[$linecount] = join(",", @line);
} elsif (($#line - 3) / 2 != $line[3]) {
print "Invalid L line at $map:$. (". (($#line - 3) / 2) ."/". $line[3] ."), correcting point count\n";
$line[3] = ($#line - 3) / 2;
$newmapfile[$linecount] = join(",", @line);
$numberofPointCountsfixed++;
$write = 1;
}
if ($line[3] == 1) {
print "Invalid L line at $map:$. only one point, line removed\n";
$numberofLlinesremoved++;
$linecount--;
$write = 1;
} elsif ($line[3] == 0) {
print "Invalid L line at $map:$. no points, line removed\n";
$numberofLlinesremoved++;
$linecount--;
$write = 1;
} else {
$newmapfile[$linecount] = join(",", @line);
}
} elsif ($line[0] eq "P") {
if ($#line != 4) {
print "Invalid P line at $map:$., line removed from file\n";
$numberofPlinesremoved++;
$linecount--;
$write = 1;
} else {
$newmapfile[$linecount] = join(",", @line);
}
} elsif ($line[0] eq "Z") {
if ($#line != 1) {
print "Invalid Z line at $map:$., line removed from file\n";
$linecount--;
$write = 1;
} else {
$newmapfile[$linecount] = join(",", @line);
}
} elsif ($line[0] eq "A") {
if ($#line != 2) {
print "Invalid A line at $map:$., line removed from file\n";
$linecount--;
$write = 1;
} else {
$newmapfile[$linecount] = join(",", @line);
}
} elsif ($line[0] eq "H") {
if ($#line != 1) {
print "Invalid H line at $map:$., line removed from file\n";
$linecount--;
$write = 1;
} else {
$newmapfile[$linecount] = join(",", @line);
}

} else {

}
}

if ($write eq 1) {
print "Old file line count = $., New file line count = ". ($#newmapfile + 1) ."\n";
print "Empty lines removed= $numberofemptylines, M lines removed = $numberofMlinesremoved, ";
print " L lines removed = $numberofLlinesremoved, Number of Point Counts fixed = $numberofPointCountsfixed\n";
print "Null's corrected = $numberofNullsfixed, P lines removed = $numberofPlinesremoved\n";
}

close(MAP);


return $write;
}

sub write_mapfile {
local $map = shift @_;

if(open(MAP, ">", $map)) {
for ($i = $#newmapfile; $i >= 0; $i--) {
print (MAP shift @newmapfile);
}
close(MAP);

print "New map file for $map writen\n\n";
} else {
print "failed to open the map file -- $map";
}
}

sub clear_array {
local $numbertopop = $#newmapfile;

for ($i = 0; $i < $numbertopop; $i++) {
pop @newmapfile;
}
}

So, Mapfiend, any possability of getting this run against your current copies of Gates of Discord maps? Or do you generate the different map formats on the fly?

If you do come across a mapfile that gets managled or does not display properly compaired to your backed up copy I would like to know so I can try and fix the script to run better.

Final warning, use at your own risk, but it should work fine or atleast it has for me.

Enjoy
~TK

[ksmith]

Originally posted by Tor K'tal
Well, I think I have it done. I haven't been able to find any files it mangaled or broke but that doesn't mean it is 100% safe. Because frankly there are too many files for me to open them all and make a visual inspection of some kind. So I would suggest backing up all your map files before running this against it. This also ONLY works against SEQ format maps. Do not attempt to use it against SOE format files.

It looks like it should work. You may want to consider generating png images of all the maps before and after and doing a simple visual comparison.


Because this is severly under-commented, I appologizes. (infact, I don't think there is a single comment in it). I recommand running it in this fassion ./sanemap.prl > sanemap.log, so that the output can be viewed later, if you care to see it. Also I do not beleive this is elligant or the most efficent possible way to do this, but it gets the job done.

Well, the original sanemap.pl wasn't commented, but I wrote it for myself.

You'd be better served by sanemap.pl | tee sanemap.log. Then you have a log and you can watch it while it runs.


A note on run time, if you aren't dumping to a log file you may think it is hung at Hateplaneb.map but that is because the files is just so huge, please calmly wait. Total run time on my system was roughtly 3 minutes (real = 2m57.472s).

Hatelaneb.map is a monstrosity. I almost wish I didn't release it so that someone would make a proper map by hand that would easily be 1/10th the size.


Note on correcting ^M's as line feeds. If the files is otherwise perfect they will not be removed, as the applications will ellect to not write a new file. Maybe Cryonic or someone could enliten me as to some logic to check if the substituion occured and thus set it to write the file (line 63).

The simple solution is to write the map every time. Consider looking at what perl -spi can do for you.

Edit: get rid of sub clear_array and use @newmapfile = (); instead.

[Tor K'tal]

Originally posted by ksmith
get rid of sub clear_array and use @newmapfile = (); instead.

Thank you very much. I couldn't figure out how to just set it to empty/null/new again, like I said, I'm very very new to perl. But, I must not be able to impliment that properly. I tried replace the clear_array(); line with what you provided and I ended up mangeling files for some reason (Chardok.map imparticular). I noticed that oldfile and newfile line counts weren't meshing anylonger as well.


Originally posted by ksmith
The simple solution is to write the map every time. Consider looking at what perl -spi can do for you.
I originally had it writing the file everytime, but then I thought if the map was okay already why take the time to write it again, under the thought process that the write time would increase the run time of the program (I'm sure I'm wrong about that). The tracking if a write needed to happen just seemed a little more friendly, and would prevent me from accidently mangaling a file if the @newmapfile array wasn't clearing properly, so I went that route. You are probably right though, it shouldn't really matter since the files are so small as far running time goes

I will look into perl -spi, thank you for that info.



Originally posted by ksmith
You'd be better served by sanemap.pl | tee sanemap.log. Then you have a log and you can watch it while it runs.

the tee was very cool, it seemed to be a little jumpy instead of scrolling smoothly across my screen and seemed to increase my run time by, oh, an entire second (increased run time, not a big deal, just thought it was interesting). But I assume that is just the fact that my linux box is pretty week as far as system specs go. So I will have to file that away in the really cool things linux can do and try and find reasons to use it. I did like it even though my box is slow and it didn't preform as I expected it. Thank you much for the new command (well new to me).



Originally posted by ksmith
It looks like it should work. You may want to consider generating png images of all the maps before and after and doing a simple visual comparison.
While I have done the png generation with PHP, I don't have a clue where to begin with perl to do that. Is there one already writen? Also if the files produce the same visual image, would a simple cmp on the two png's let me know if they were different? My conceren is that if the color of the line changed on the null files it would produces a false possative for different files. But like you said, a visual inspection would eleviate that.


Overall, thank you for the information and feed back.

~ TK

[ksmith]

Originally posted by Tor K'tal
While I have done the png generation with PHP, I don't have a clue where to begin with perl to do that. Is there one already writen? Also if the files produce the same visual image, would a simple cmp on the two png's let me know if they were different? My conceren is that if the color of the line changed on the null files it would produces a false possative for different files. But like you said, a visual inspection would eleviate that.

Check my sig for links to where I discuss generating png/gif files from showeq maps. One of those threads has a link to the script I wrote. Also, Darkgrue made a somewhat more friendly version if I recall correctly. Currently, my pngmap.pl script uses code based on sanemap.pl when loading the map file, so for maps with (null) lines, it would just print an error and not generate a map. A visual inspection of all the images after being read through the cleanup script would probably be the simplest solution.

[Mapfiend]

I complained when SOE first introduced their map format, wondering what in the hell could posess them to put only one map segment per line. The answer is quite simple: MUCH faster parsing and rendering.
I recently converted my png generator to use SOE maps instead of SEQ maps, and the difference is amazing. It used to take on the order of 5 mins to generate a PNG for poknowledge. Using the SOE map, it takes just under 15 secs.
I've also written a script to convert SOE maps into PDF format (using pdflib) and I can convert ALL the maps on my site (653 atm) in under 3 mins on my local machine.