Looking for a PGN/EPD utility

Discussion of anything and everything relating to chess playing software and machines.

Moderators: Harvey Williamson, bob, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Dann Corbit
Posts: 11622
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Looking for a PGN/EPD utility

Post by Dann Corbit » Tue Sep 08, 2015 5:07 am

I am looking for a utility to process a PGN file.

Specifically, I want to provide a list of EPD positions and remove all the games in the file that have that position.

Equally good would be a utility that can create a new file with all of the games that contain a position from the EPD list.

I am not asking for someone to write this. I can do the job with (for instance) Scid-vs-Pc by setting the EPD positions one at a time and deleting the games. But perhaps someone familiar with what is already available know of such a utility.

User avatar
Ozymandias
Posts: 1222
Joined: Sun Oct 25, 2009 12:30 am

Re: Looking for a PGN/EPD utility

Post by Ozymandias » Tue Sep 08, 2015 8:17 am

I'm going to be needing something like that, a few months down the road. Let's see if someone can spot which one, of the myriad of PGN tools out there, can perform such a task.

Norm Pollock
Posts: 1029
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Norm Pollock » Tue Sep 08, 2015 12:21 pm

If you search for an epd using all 4 fields of the epd, you have to be careful about false "en passant" notations. They could cause the search to miss if one epd record has a false ep and the other just has "-", or if they have different false ep.

User avatar
Ozymandias
Posts: 1222
Joined: Sun Oct 25, 2009 12:30 am

Re: Looking for a PGN/EPD utility

Post by Ozymandias » Tue Sep 08, 2015 5:27 pm

Norm Pollock wrote:If you search for an epd using all 4 fields of the epd, you have to be careful about false "en passant" notations. They could cause the search to miss if one epd record has a false ep and the other just has "-", or if they have different false ep.
Could an epd record of the final position, as produced by epdFin, include false "en passant" information? Assuming the answer is "no", how would one proceed to find games, with that position, in another PGN file?

Ferdy
Posts: 4293
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Looking for a PGN/EPD utility

Post by Ferdy » Tue Sep 08, 2015 6:30 pm

Dann Corbit wrote:I am looking for a utility to process a PGN file.

Specifically, I want to provide a list of EPD positions and remove all the games in the file that have that position.

Equally good would be a utility that can create a new file with all of the games that contain a position from the EPD list.

I am not asking for someone to write this. I can do the job with (for instance) Scid-vs-Pc by setting the EPD positions one at a time and deleting the games. But perhaps someone familiar with what is already available know of such a utility.
Try pgn-extract this way.

1. Create tag.txt where it contains fen lines based from your epd file

Code: Select all

FEN "rnbqkb1r/pp2pppp/3p1n2/8/3NP3/8/PPP2PPP/RNBQKB1R w KQkq - 0 1"
FEN "rnbqkbnr/ppp1pppp/8/3p4/2PP4/8/PP2PPPP/RNBQKBNR b KQkq - 0 1"
[...]
2. Extract games by running pgn-extract with following command lines.

From batch file e.bat

Code: Select all

:: (1) extract games when all pos in a game is not found in all pos from epd file
pgn-extract-17-21 -s -ttag.txt -nsrc-out-notfound-in-epd.pgn src.pgn


:: (2) extract games when at least a pos in a game is found in any pos found in epd file
pgn-extract-17-21 -s -ttag.txt -osrc-out-found-in-epd.pgn src.pgn
src.pgn is your input file, this is not revised by pgn-extract.
-ttag.txt, option -t to read tag.txt
-nsrc-out-notfound-in-epd.pgn, using -n option will output games if the pos in this game is not in the epd file. The output file is src-out-notfound-in-epd.pgn.
-osrc-out-found-in-epd.pgn, using -o option the normal way of outputting what is in the epd file (overwrite mode). The output filename is src-out-found-in-epd.pgn.

Sample script in python 2.7... to create tags, just in case you need it.
Even if your epd has more than 4 fields with op codes, this script will only consider the first 4 fields.
Then it will add 0 and 1 for hmvc and fmvn since pgn-extract will give warning when using FEN tag without hmvc and fmvn. Use the latest pgn-extract-17-21 as it has some bug fixes and improvements.
Sample output in tag.txt.

Code: Select all

FEN "rnbqkb1r/pp2pppp/3p1n2/8/3NP3/8/PPP2PPP/RNBQKB1R w KQkq - 0 1"
create_tag.py. You need to rename your epd file to 1.epd.

Code: Select all

import sys
import os

    
def delete_file(fn):
    """ Delete file fn """
    if os.path.isfile(fn):
        os.remove(fn)


def creat_tag(in_epd_fn, out_tag):
    """ read epd file and create out_tag """
    # Delete existing tag file
    delete_file(out_tag)
    
    with open(in_epd_fn, 'r') as f:
        for lines in f:
            line = lines.strip()
            splitted_epd = line.split(' ')

            # Take the first 4 fields only
            epd = ' '.join(splitted_epd[:4])

            # Save the epd line with different format
            with open(out_tag, 'a') as fo:
                fo.write('FEN "%s 0 1"\n' % epd)

    
def main(argv):
    """ starts at main """    
    # Vars
    epd_fn = '1.epd'
    tag_fn = 'tag.txt'

    # check epd file 1.epd is existing
    if not os.path.isfile(epd_fn):
        print 'process stopped, %s file is missing' % epd_fn
        raw_input('Press enter key to exit')
        sys.exit(1)

    # Create tag.txt
    creat_tag(epd_fn, tag_fn)

    print 'done!!'
    raw_input('Press enter key to exit')
    

if __name__ == "__main__":
    main(sys.argv[1:])

Dann Corbit
Posts: 11622
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Dann Corbit » Tue Sep 08, 2015 7:42 pm

Norm Pollock wrote:If you search for an epd using all 4 fields of the epd, you have to be careful about false "en passant" notations. They could cause the search to miss if one epd record has a false ep and the other just has "-", or if they have different false ep.
I always strip off false e.p. fields. It saves me a lot of analysis time. However, if the tool that is looking for the PGN wants the field to be there, then it is a problem.
;-)

Supposedly, this is 9 different game positions (if we toss in all 8 irrelevant e.p. fields):
[d]rnbqkbnr/8/8/pppppppp/PPPPPPPP/8/8/RNBQKBNR w KQkq -
Same thing, if we change the side to move to black.

Clearly a defect in the original specification.

User avatar
Ozymandias
Posts: 1222
Joined: Sun Oct 25, 2009 12:30 am

Re: Looking for a PGN/EPD utility

Post by Ozymandias » Tue Sep 08, 2015 10:08 pm

That should work, thx.

Dann Corbit
Posts: 11622
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Dann Corbit » Tue Sep 08, 2015 10:16 pm

This looks like the solution.
That you ever so much.
Saved me from having to write something to do the same job.

Norm Pollock
Posts: 1029
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Looking for a PGN/EPD utility

Post by Norm Pollock » Wed Sep 09, 2015 1:46 am

I suggest that whenever anyone uses PGN-Extract for creating an epd version of a pgn description, to use the command "--nofauxep". This will ensure that no false en passant notation is output to the epd file. This command is new to the latest version, 17-21.

User avatar
Ozymandias
Posts: 1222
Joined: Sun Oct 25, 2009 12:30 am

Re: Looking for a PGN/EPD utility

Post by Ozymandias » Wed Sep 09, 2015 8:01 am

Norm Pollock wrote:This command is new to the latest version, 17-21.
From pgn-extract's change history: "after a suggestion by Norm Pollock". So, that answers my previous question.

Post Reply