Looking for a PGN/EPD utility

[Norm Pollock]

http://www.mediafire.com/download/7xkuc ... sition.exe

I set "size" to 100000 from 100

100000 is going to take awhile to process.

I suggest putting a number comment in each game of the pgn file if you have not already done so. One such program that does this is "gameNum". It is in the 40H-PGN suite on my home page (www below).

I finally tried epdPostion and I still get the same message (Array Index Out of Bounds error!), only at 100001 lines, of course. The search.epd file is 730 MB, and the temp.epdn is 1.11 GB.

Can we amp up the steroids again (3 more zeros)?

Juan--

Changed 100000 to 100000000

http://www.mediafire.com/file/n76acc3sw ... teroid.exe

this is the executable, no need to unpack it
use it the same as "epdPosition" except the name is "epdPositionSteroid"

if it works, please let me know the capacities of your machine.

-Norm

[Ozymandias]

if it works, please let me know the capacities of your machine.

-Norm

Thx for the quick reply, it didn't work, I get a window saying: Too many heap sections (Fatal error in gc).

5820K, Win8 64, 16 GB

[Norm Pollock]

took out one "0", and added output line to show line limit for search.epd

line limit now 10000000

same link

http://www.mediafire.com/file/n76acc3sw ... teroid.exe

good luck

-Norm

[Ozymandias]

took out one "0"

I'm making the search.epd smaller, but still the same gc error. Maybe the original epdPosition was already at the limit? At least with that one I simply got an out of array error.

[Norm Pollock]

search limit now down to 1000000

If that does not work you could split search.epd using "txtSplit" with 40H-TXT

same link as above

[Ozymandias]

search limit now down to 1000000

If that does not work you could split search.epd using "txtSplit" with 40H-TXT

same link as above

That's probably a sensible number, and a reasonable limit. I've had it running for hours now, with a search.epd of less than a million lines, but I'm still waiting to see if the result is ok. I'm using the second version of epdPositionSteroid you posted, but it'd probably make no practical difference.

Thx for the heads up on txtSplit, the first of the suite I'm putting to good use.

[Norm Pollock]

Good to hear it is at least running. Hope the output is ok.

I will put a clone of "txtSplit" ("epdSplit") into 40H-EPD next go-around. Likewise for "txtSort".

[Ozymandias]

Hope the output is ok.

Doesn't look that way, from what I could sample (still running):

Code: Select all

3rr1k1/1bqn1ppp/p2b4/2pP4/Pp6/4BN1P/1PQ1BPP1/2RR2K1 w - -
game = 1    after ply 503215

r4rk1/1q1n1ppp/p4n2/4pN2/1PP1p3/4bP2/3QB1PP/R4RK1 w - -
game = 1    after ply 503635

r4k2/3b1pp1/p3p2p/b1np4/8/2P5/PP2BPPP/2K2R1R w - -
game = 1    after ply 503659

2rq1rk1/1b1nbppp/p3p3/8/PpN1P3/3BB3/1P2QPPP/2R2RK1 w - -
game = 1    after ply 503805

r4rk1/2p2ppp/p7/1pB5/2p2P2/2nq1QPB/P7/R4RK1 w - -
game = 1    after ply 504294

1q3rk1/3n1ppp/4pn2/1N6/1p1P4/1Q3B2/1P1B1PPb/R6K w - -
game = 1    after ply 504418

4r1k1/1r2bppp/pqNp4/1p1Pp1Pn/7P/P4P2/1PPQ3R/2KR1B2 w - -
game = 1    after ply 504766

r2q1rk1/pp2nppp/8/2n1b1PP/2BN2b1/P1N1P3/2QB1P2/R3K2R w KQ -
game = 1    after ply 504928

r3k2r/2pb1pp1/pq5p/1p2P3/2p2P2/2P3P1/P3Q1BP/3R1RK1 w kq -
game = 1    after ply 505753

It goes on like that. For one, I used gameNum in the PGN that I want to finally process, and there's no way that more than 3 positions are found in the same game. And the ply counts are ridiculous, games are truncated at ply 40, both in the PGN that produced the search.epd as well as the final one.

[Ozymandias]

Actually, I tried with a much smaller search.epd and the original epdPosition. I see the same pattern. This is the bat file, up to that point, is there something wrong?

Code: Select all

pgn-extract -Wepd --nofauxep -s -otemp.epd set1a20.pgn

epdFin temp.epd

del temp.epd

rename outF.epd search.epd

gameNum pyramid_ply40.pgn

pgn-extract -Wepd --nofauxep -s -otemp.epd out4.pgn

epdPosition search.epd temp.epd

[Norm Pollock]

Hope the output is ok.

Doesn't look that way, from what I could sample (still running):

Code: Select all

3rr1k1/1bqn1ppp/p2b4/2pP4/Pp6/4BN1P/1PQ1BPP1/2RR2K1 w - -
game = 1    after ply 503215

r4rk1/1q1n1ppp/p4n2/4pN2/1PP1p3/4bP2/3QB1PP/R4RK1 w - -
game = 1    after ply 503635

r4k2/3b1pp1/p3p2p/b1np4/8/2P5/PP2BPPP/2K2R1R w - -
game = 1    after ply 503659

2rq1rk1/1b1nbppp/p3p3/8/PpN1P3/3BB3/1P2QPPP/2R2RK1 w - -
game = 1    after ply 503805

r4rk1/2p2ppp/p7/1pB5/2p2P2/2nq1QPB/P7/R4RK1 w - -
game = 1    after ply 504294

1q3rk1/3n1ppp/4pn2/1N6/1p1P4/1Q3B2/1P1B1PPb/R6K w - -
game = 1    after ply 504418

4r1k1/1r2bppp/pqNp4/1p1Pp1Pn/7P/P4P2/1PPQ3R/2KR1B2 w - -
game = 1    after ply 504766

r2q1rk1/pp2nppp/8/2n1b1PP/2BN2b1/P1N1P3/2QB1P2/R3K2R w KQ -
game = 1    after ply 504928

r3k2r/2pb1pp1/pq5p/1p2P3/2p2P2/2P3P1/P3Q1BP/3R1RK1 w kq -
game = 1    after ply 505753

It goes on like that. For one, I used gameNum in the PGN that I want to finally process, and there's no way that more than 3 positions are found in the same game. And the ply counts are ridiculous, games are truncated at ply 40, both in the PGN that produced the search.epd as well as the final one.

Wow! I ran a similar project to yours with 15K games in the file for search.epd, and 47K games in the input pgn file. All ran fine.

Based on the output of "manifest-ps" that you sent me, my first suspicion is that the "pgn" files you used were not up to PGN Standards. Perhaps they did not have a blank line separating games.

Please use "trim" on those files and then rename.
trim alpha.pgn
outr.pgn alpha-t.pgn

I am surprised that you were able to get output from manifest-ps during execution. It must have been due to an overflow of the buffer because during execution of my test the files "manifest-ps" and "numbers", the two output files of epdPosition, had 0 bytes.

Based on the size of your test, it will take a very long time for it to finish. Perhaps a month. In which time there could be a power blip that wipes everything out. I would do the project in small parts.

Also I would rename "out4.pgn" before re-using it.
gameNum alpha.pgn
ren out4.pgn alpha-n.pgn

And finally, are you using a recent version of pgn-extract? Use "17-21" or later. I noticed D. Barnes' new version "17-30" outputs "17-26" when using the "--version" command.

-Norm