Page 1 of 4

Looking for a PGN/EPD utility

Posted: Tue Sep 08, 2015 7:07 am
by Dann Corbit
I am looking for a utility to process a PGN file.

Specifically, I want to provide a list of EPD positions and remove all the games in the file that have that position.

Equally good would be a utility that can create a new file with all of the games that contain a position from the EPD list.

I am not asking for someone to write this. I can do the job with (for instance) Scid-vs-Pc by setting the EPD positions one at a time and deleting the games. But perhaps someone familiar with what is already available know of such a utility.

Re: Looking for a PGN/EPD utility

Posted: Tue Sep 08, 2015 10:17 am
by Ozymandias
I'm going to be needing something like that, a few months down the road. Let's see if someone can spot which one, of the myriad of PGN tools out there, can perform such a task.

Re: Looking for a PGN/EPD utility

Posted: Tue Sep 08, 2015 2:21 pm
by Norm Pollock
If you search for an epd using all 4 fields of the epd, you have to be careful about false "en passant" notations. They could cause the search to miss if one epd record has a false ep and the other just has "-", or if they have different false ep.

Re: Looking for a PGN/EPD utility

Posted: Tue Sep 08, 2015 7:27 pm
by Ozymandias
Norm Pollock wrote:If you search for an epd using all 4 fields of the epd, you have to be careful about false "en passant" notations. They could cause the search to miss if one epd record has a false ep and the other just has "-", or if they have different false ep.
Could an epd record of the final position, as produced by epdFin, include false "en passant" information? Assuming the answer is "no", how would one proceed to find games, with that position, in another PGN file?

Re: Looking for a PGN/EPD utility

Posted: Tue Sep 08, 2015 8:30 pm
by Ferdy
Dann Corbit wrote:I am looking for a utility to process a PGN file.

Specifically, I want to provide a list of EPD positions and remove all the games in the file that have that position.

Equally good would be a utility that can create a new file with all of the games that contain a position from the EPD list.

I am not asking for someone to write this. I can do the job with (for instance) Scid-vs-Pc by setting the EPD positions one at a time and deleting the games. But perhaps someone familiar with what is already available know of such a utility.
Try pgn-extract this way.

1. Create tag.txt where it contains fen lines based from your epd file

Code: Select all

FEN "rnbqkb1r/pp2pppp/3p1n2/8/3NP3/8/PPP2PPP/RNBQKB1R w KQkq - 0 1"
FEN "rnbqkbnr/ppp1pppp/8/3p4/2PP4/8/PP2PPPP/RNBQKBNR b KQkq - 0 1"
[...]
2. Extract games by running pgn-extract with following command lines.

From batch file e.bat

Code: Select all

:: (1) extract games when all pos in a game is not found in all pos from epd file
pgn-extract-17-21 -s -ttag.txt -nsrc-out-notfound-in-epd.pgn src.pgn


:: (2) extract games when at least a pos in a game is found in any pos found in epd file
pgn-extract-17-21 -s -ttag.txt -osrc-out-found-in-epd.pgn src.pgn
src.pgn is your input file, this is not revised by pgn-extract.
-ttag.txt, option -t to read tag.txt
-nsrc-out-notfound-in-epd.pgn, using -n option will output games if the pos in this game is not in the epd file. The output file is src-out-notfound-in-epd.pgn.
-osrc-out-found-in-epd.pgn, using -o option the normal way of outputting what is in the epd file (overwrite mode). The output filename is src-out-found-in-epd.pgn.

Sample script in python 2.7... to create tags, just in case you need it.
Even if your epd has more than 4 fields with op codes, this script will only consider the first 4 fields.
Then it will add 0 and 1 for hmvc and fmvn since pgn-extract will give warning when using FEN tag without hmvc and fmvn. Use the latest pgn-extract-17-21 as it has some bug fixes and improvements.
Sample output in tag.txt.

Code: Select all

FEN "rnbqkb1r/pp2pppp/3p1n2/8/3NP3/8/PPP2PPP/RNBQKB1R w KQkq - 0 1"
create_tag.py. You need to rename your epd file to 1.epd.

Code: Select all

import sys
import os

    
def delete_file(fn):
    """ Delete file fn """
    if os.path.isfile(fn):
        os.remove(fn)


def creat_tag(in_epd_fn, out_tag):
    """ read epd file and create out_tag """
    # Delete existing tag file
    delete_file(out_tag)
    
    with open(in_epd_fn, 'r') as f:
        for lines in f:
            line = lines.strip()
            splitted_epd = line.split(' ')

            # Take the first 4 fields only
            epd = ' '.join(splitted_epd[:4])

            # Save the epd line with different format
            with open(out_tag, 'a') as fo:
                fo.write('FEN "%s 0 1"\n' % epd)

    
def main(argv):
    """ starts at main """    
    # Vars
    epd_fn = '1.epd'
    tag_fn = 'tag.txt'

    # check epd file 1.epd is existing
    if not os.path.isfile(epd_fn):
        print 'process stopped, %s file is missing' % epd_fn
        raw_input('Press enter key to exit')
        sys.exit(1)

    # Create tag.txt
    creat_tag(epd_fn, tag_fn)

    print 'done!!'
    raw_input('Press enter key to exit')
    

if __name__ == "__main__":
    main(sys.argv[1:])

Re: Looking for a PGN/EPD utility

Posted: Tue Sep 08, 2015 9:42 pm
by Dann Corbit
Norm Pollock wrote:If you search for an epd using all 4 fields of the epd, you have to be careful about false "en passant" notations. They could cause the search to miss if one epd record has a false ep and the other just has "-", or if they have different false ep.
I always strip off false e.p. fields. It saves me a lot of analysis time. However, if the tool that is looking for the PGN wants the field to be there, then it is a problem.
;-)

Supposedly, this is 9 different game positions (if we toss in all 8 irrelevant e.p. fields):
[d]rnbqkbnr/8/8/pppppppp/PPPPPPPP/8/8/RNBQKBNR w KQkq -
Same thing, if we change the side to move to black.

Clearly a defect in the original specification.

Re: Looking for a PGN/EPD utility

Posted: Wed Sep 09, 2015 12:08 am
by Ozymandias
That should work, thx.

Re: Looking for a PGN/EPD utility

Posted: Wed Sep 09, 2015 12:16 am
by Dann Corbit
This looks like the solution.
That you ever so much.
Saved me from having to write something to do the same job.

Re: Looking for a PGN/EPD utility

Posted: Wed Sep 09, 2015 3:46 am
by Norm Pollock
I suggest that whenever anyone uses PGN-Extract for creating an epd version of a pgn description, to use the command "--nofauxep". This will ensure that no false en passant notation is output to the epd file. This command is new to the latest version, 17-21.

Re: Looking for a PGN/EPD utility

Posted: Wed Sep 09, 2015 10:01 am
by Ozymandias
Norm Pollock wrote:This command is new to the latest version, 17-21.
From pgn-extract's change history: "after a suggestion by Norm Pollock". So, that answers my previous question.