Looking for a PGN/EPD utility

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Norm Pollock
Posts: 1029
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Norm Pollock » Wed Sep 09, 2015 6:17 pm

I have written a program "epdPosition" that together in a batch file with "PGN Extract 17-21" and one of my other programs "numExtract", should meet the specs mentioned by Dann Corbit in the first post.

download (containing epdPosition.exe, numExtract.exe and pgnPosition.bat):

http://www.mediafire.com/download/5g5tg ... osition.7z

The batch file is pgnPosition.bat

It uses a user-made text file ("positions") of epd positions, 1 per line

Usage:

pgnPosition positions alpha.pgn

(make sure all 3 programs are accessible)

Output:

manifest-ps (listing of each match: game #, ply #)
outZ.pgn (extracted games that contain the match)
excludeZ.pgn (extracted games NOT containing a match)

Contents of pgnPosition.bat

pgn-extract -Wepd --nofauxep -s -otemp.epd %2
epdPosition %1 temp.epd
numExtract numbers %2

Note: a game can contain more that one match

Dann Corbit
Posts: 11512
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Dann Corbit » Wed Sep 09, 2015 6:21 pm

Fabulous.
Thank you for this.
It will save me a lot of time.

Dann Corbit
Posts: 11512
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Dann Corbit » Sat Sep 12, 2015 4:47 am

It appears that this process allows only 100 lines from the EPD file:

F:\chess\position>pgn-extract -Wepd --nofauxep -s -otemp.epd iccfb.pgn
Games: 9010
F:\chess\position>epdPosition positions temp.epd
Array Index Out of Bounds error!
line number 101
FEN "rn3rk1/ppq2pp1/2pbpn1p/8/N1PP1Bb1/5NP1/PP3PBP/R2QR1K1 w - - 0 1"

F:\chess\position>numextract numbers iccfb.pgn

Number of games in iccfb.pgn is 9016

Number of games in outZ.pgn is 0

finished: be sure to rename/copy outZ.pgn and excludeZ.pgn


I have 2078 in this particular file, and I am likely to have tens of thousands in other files. Is it possible to get the array size configurable or allocated to a large size such as 100,000?

Norm Pollock
Posts: 1029
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Norm Pollock » Sat Sep 12, 2015 5:20 am

http://www.mediafire.com/download/7xkuc ... sition.exe

I set "size" to 100000 from 100

100000 is going to take awhile to process.

I suggest putting a number comment in each game of the pgn file if you have not already done so. One such program that does this is "gameNum". It is in the 40H-PGN suite on my home page (www below).

User avatar
Ozymandias
Posts: 1214
Joined: Sun Oct 25, 2009 12:30 am

Re: Looking for a PGN/EPD utility

Post by Ozymandias » Sat Sep 12, 2015 5:42 am

Norm Pollock wrote:http://www.mediafire.com/download/7xkuc ... sition.exe

I set "size" to 100000 from 100

100000 is going to take awhile to process.

I suggest putting a number comment in each game of the pgn file if you have not already done so. One such program that does this is "gameNum". It is in the 40H-PGN suite on my home page (www below).
Does "epdFin" have any such limitation? I'm going to need, to run millions of positions, trough it.

Dann Corbit
Posts: 11512
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Dann Corbit » Sat Sep 12, 2015 6:05 am

Are the elements of file positions supposed to be of this format:

FEN "1rb2rk1/1p1nqpbp/p2p1np1/2pP4/P4B2/2N1PN1P/1P2BPP1/R2Q1RK1 w - - 0 1"

or something else?
I get wrong filtering.

Edit:
I figured it out.
You wanted:
rr4k1/3nppbp/b2p1np1/q1pP4/8/2N2NP1/PPQ1PPBP/1RB2RK1 w - - 0 1

Norm Pollock
Posts: 1029
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Norm Pollock » Sat Sep 12, 2015 6:08 am

Ozymandias wrote:
Norm Pollock wrote:http://www.mediafire.com/download/7xkuc ... sition.exe

I set "size" to 100000 from 100

100000 is going to take awhile to process.

I suggest putting a number comment in each game of the pgn file if you have not already done so. One such program that does this is "gameNum". It is in the 40H-PGN suite on my home page (www below).
Does "epdFin" have any such limitation? I'm going to need, to run millions of positions, trough it.
No. You should be fine. but there is a limit that no single game in the original "pgn" file can have more than 2000 lines (positions). If you exceed that, then there will be a crash. 2000 lines = 999.5 moves. Let me know if you hit that limit.

I am reviewing all my programs to find those that use a user-specified file that gets read into an array. Those files are limited by the size of the array. Most likely I will enlarge the array size.

Norm Pollock
Posts: 1029
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Norm Pollock » Sat Sep 12, 2015 6:22 am

Dann Corbit wrote:Are the elements of file positions supposed to be of this format:

FEN "1rb2rk1/1p1nqpbp/p2p1np1/2pP4/P4B2/2N1PN1P/1P2BPP1/R2Q1RK1 w - - 0 1"

or something else?
I get wrong filtering.

Edit:
I figured it out.
You wanted:
rr4k1/3nppbp/b2p1np1/q1pP4/8/2N2NP1/PPQ1PPBP/1RB2RK1 w - - 0 1
The "epd" record can have a comment, or be a "FEN". It's only the first 4 fields that are matched. Examples:

rr4k1/3nppbp/b2p1np1/q1pP4/8/2N2NP1/PPQ1PPBP/1RB2RK1 w - - 0 1
rr4k1/3nppbp/b2p1np1/q1pP4/8/2N2NP1/PPQ1PPBP/1RB2RK1 w - - c0 comment;

User avatar
Ozymandias
Posts: 1214
Joined: Sun Oct 25, 2009 12:30 am

Re: Looking for a PGN/EPD utility

Post by Ozymandias » Sat Sep 12, 2015 9:32 am

Norm Pollock wrote:there is a limit that no single game in the original "pgn" file can have more than 2000 lines (positions). If you exceed that, then there will be a crash. 2000 lines = 999.5 moves. Let me know if you hit that limit.
I truncate games at ply 600, so I don't anticipate any problems.

User avatar
Ozymandias
Posts: 1214
Joined: Sun Oct 25, 2009 12:30 am

Re: Looking for a PGN/EPD utility

Post by Ozymandias » Wed Feb 15, 2017 9:30 am

Norm Pollock wrote:http://www.mediafire.com/download/7xkuc ... sition.exe

I set "size" to 100000 from 100

100000 is going to take awhile to process.

I suggest putting a number comment in each game of the pgn file if you have not already done so. One such program that does this is "gameNum". It is in the 40H-PGN suite on my home page (www below).
I finally tried epdPostion and I still get the same message (Array Index Out of Bounds error!), only at 100001 lines, of course. The search.epd file is 730 MB, and the temp.epdn is 1.11 GB.

Can we amp up the steroids again (3 more zeros)?

Post Reply