Chess Tools

[Ferdy]

Hi Ferdy,
Big Fan of your work and chess tools.
If this thread is still active, I would like to make a request.
Recently, I have been collecting epd testsuites; however, testsuites are notoriously incestuous, meaning they almost always include identical positions often with different ids.

In contrast, managing pgn databases is a bit easier to remove duplicate games using tools such as pgn-extract, but I have not found any similar tool to manage epd databases (specifically removing duplicate fen positions from epd files).

Right now, the easiest way to accomplish this is to strip all op codes from a database (using scidvspc tools) and then use a simple awk expression in terminal to remove duplicate lines: awk '!seen[$0]++' Input.epd > Output.epd
But this simply perpetuates the same problem I am describing, because I end up prescribing a different ID to positions already accounted for.

Could you create a tool to manage epd databases, that parses for the FEN of each epd position, and remove the lines of any subsequent duplicates?

Given epd's
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm e4; id "my test 1";
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm d4; id "my test 2";

Are the two epd's above identical? Perhaps not.

But this one has, and 2nd epd can be removed.
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm e4; id "my test 1";
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm e4; id "my test 2";

What if there are other opcodes? But it seems like [piece locations] [stm] [castle right] [ep square] bm [move] can be considered as independent and can be used to identify identical, depends on your criteria too.

I think you need to give me some examples which epd's are considered identical.

Hi Ferdy,

Right, I have previously found situations like this before in my database, where identical positions had different best moves. So I am asking for an imperfect solution, and ignore the bm/am/pm op code. Here is an example I found in various sources I pulled together:

Example Pooled EPD File (i.e. Input):

line 1: 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - bm Bh5+; id "W.Eigenmann: Brillant 002";
line 2 : 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - id ?; bm Bh5;
line 3 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id "arasan19.45"; c0 "Camacho Martinez-An. C. Hernandez, Cuba 1995";
line 4 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id peg124;

Desired Output:

line 1: 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - bm Bh5+; id "W.Eigenmann: Brillant 002";
line 2 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id "arasan19.45"; c0 "Camacho Martinez-An. C. Hernandez, Cuba 1995";

And If the lines in the example input file were reversed, then lines 4 and 2 would be printed instead:
line 1 : 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - id ?; bm Bh5;
line 2 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id peg124;

Please note that there are also situations with more than two replicates, for example:
8/2p1p1pp/2P1P2k/6pP/p7/P4Bp1/1P4P1/n5K1 w - - bm Bd1; id "MES.256";
8/2p1p1pp/2P1P2k/6pP/p7/P4Bp1/1P4P1/n5K1 w - - bm Bd1; id "Holmes Endgame Pos. 1362"; c0 "white pieces=8 black pieces=9"; c1 "material balance: -1,0";
8/2p1p1pp/2P1P2k/6pP/p7/P4Bp1/1P4P1/n5K1 w - - bm Bd1; c0 "level: med-3"; id "EG 0866";

This is much better, just select the first entry if there are more identical.
Identical can be defined to be same piece locations, same side to move, same castling rights, same ep square and same bm, ignore other op codes.

Do you use windows OS?, Do you use python 2 or 3?

[Spill_The_Tea]

Hi Ferdy,
Big Fan of your work and chess tools.
If this thread is still active, I would like to make a request.
Recently, I have been collecting epd testsuites; however, testsuites are notoriously incestuous, meaning they almost always include identical positions often with different ids.

In contrast, managing pgn databases is a bit easier to remove duplicate games using tools such as pgn-extract, but I have not found any similar tool to manage epd databases (specifically removing duplicate fen positions from epd files).

Right now, the easiest way to accomplish this is to strip all op codes from a database (using scidvspc tools) and then use a simple awk expression in terminal to remove duplicate lines: awk '!seen[$0]++' Input.epd > Output.epd
But this simply perpetuates the same problem I am describing, because I end up prescribing a different ID to positions already accounted for.

Could you create a tool to manage epd databases, that parses for the FEN of each epd position, and remove the lines of any subsequent duplicates?

Given epd's
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm e4; id "my test 1";
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm d4; id "my test 2";

Are the two epd's above identical? Perhaps not.

But this one has, and 2nd epd can be removed.
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm e4; id "my test 1";
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - bm e4; id "my test 2";

What if there are other opcodes? But it seems like [piece locations] [stm] [castle right] [ep square] bm [move] can be considered as independent and can be used to identify identical, depends on your criteria too.

I think you need to give me some examples which epd's are considered identical.

Hi Ferdy,

Right, I have previously found situations like this before in my database, where identical positions had different best moves. So I am asking for an imperfect solution, and ignore the bm/am/pm op code. Here is an example I found in various sources I pulled together:

Example Pooled EPD File (i.e. Input):

line 1: 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - bm Bh5+; id "W.Eigenmann: Brillant 002";
line 2 : 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - id ?; bm Bh5;
line 3 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id "arasan19.45"; c0 "Camacho Martinez-An. C. Hernandez, Cuba 1995";
line 4 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id peg124;

Desired Output:

line 1: 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - bm Bh5+; id "W.Eigenmann: Brillant 002";
line 2 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id "arasan19.45"; c0 "Camacho Martinez-An. C. Hernandez, Cuba 1995";

And If the lines in the example input file were reversed, then lines 4 and 2 would be printed instead:
line 1 : 8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - id ?; bm Bh5;
line 2 : 8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id peg124;

Please note that there are also situations with more than two replicates, for example:
8/2p1p1pp/2P1P2k/6pP/p7/P4Bp1/1P4P1/n5K1 w - - bm Bd1; id "MES.256";
8/2p1p1pp/2P1P2k/6pP/p7/P4Bp1/1P4P1/n5K1 w - - bm Bd1; id "Holmes Endgame Pos. 1362"; c0 "white pieces=8 black pieces=9"; c1 "material balance: -1,0";
8/2p1p1pp/2P1P2k/6pP/p7/P4Bp1/1P4P1/n5K1 w - - bm Bd1; c0 "level: med-3"; id "EG 0866";

This is much better, just select the first entry if there are more identical.
Identical can be defined to be same piece locations, same side to move, same castling rights, same ep square and same bm, ignore other op codes.

Do you use windows OS?, Do you use python 2 or 3?

I am on MacOS, so a python script is ideal. Since the python-chess module no longer supports python 2, I figure python 3 is a bit better in principle. But I have both, so whatever is easier.

Edit: I reread your response, and I want to define identical as the same side to move, same castling rights, and same ep square. And exclude bm/am/pm and all other op codes. The reason, is because epd format isn't always correctly adhered to. Sometimes, people place the principle variation in the bm op code, or show multiple best moves with different scoring metric. Consider the following example:

rnbqkb1r/1p2pppp/p2p4/6B1/3NP1n1/2N5/PPP2PPP/R2QKB1R b KQkq - bm h6; c0 "Analysis by Rybkav2.3.2a.mp.x64 @300sec@3.0GHz@4threads W512MB hash"; bm Nc6 {4} h6 {415};

rnbqkb1r/1p2pppp/p2p4/6B1/3NP1n1/2N5/PPP2PPP/R2QKB1R b KQkq - bm h6; c0 "h6(23)";

[Ferdy]

I am on MacOS, so a python script is ideal. Since the python-chess module no longer supports python 2, I figure python 3 is a bit better in principle. But I have both, so whatever is easier.

Edit: I reread your response, and I want to define identical as the same side to move, same castling rights, and same ep square. And exclude bm/am/pm and all other op codes. The reason, is because epd format isn't always correctly adhered to. Sometimes, people place the principle variation in the bm op code, or show multiple best moves with different scoring metric. Consider the following example:

rnbqkb1r/1p2pppp/p2p4/6B1/3NP1n1/2N5/PPP2PPP/R2QKB1R b KQkq - bm h6; c0 "Analysis by Rybkav2.3.2a.mp.x64 @300sec@3.0GHz@4threads W512MB hash"; bm Nc6 {4} h6 {415};

rnbqkb1r/1p2pppp/p2p4/6B1/3NP1n1/2N5/PPP2PPP/R2QKB1R b KQkq - bm h6; c0 "h6(23)";

Code: Select all

py -3 remove_iden_epd.py -h
usage: Identical epd remover v0.1.0 [-h] -i INPUT -o OUTPUT

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        input epd filename
  -o OUTPUT, --output OUTPUT
                        output filename, overwrite mode

This one does not need python-chess module since we are not checking the bm move legality and correcting legality of the whole epd line.

Try this code, remove_iden_epd.py

Code: Select all

# -*- coding: utf-8 -*-
"""
remove identical epd

Requirements:
    Python 3

"""

import argparse


VERSION = 'v0.1.0'


def main():
    parser = argparse.ArgumentParser(prog='Identical epd remover {}'.format(VERSION))
    parser.add_argument('-i', '--input', help='input epd filename',
                        required=True)
    parser.add_argument('-o', '--output', help='output filename, overwrite mode',
                        required=True)
    
    args = parser.parse_args()    
    epdfn = args.input
    outfn = args.output
    
    tmp_save = []
    uni_save = []
    
    # Read input epd and remove identical, consider first entry as priority
    with open(epdfn) as f:
        for lines in f:
            line = lines.strip()
            
            base_epd = ' '.join(line.split()[0:4]).strip()           
            if base_epd not in tmp_save:
                tmp_save.append(base_epd)
                uni_save.append(line)

    # Write to output file
    with open(outfn, 'w') as f:
        for epd_line in uni_save:
            f.write('{}\n'.format(epd_line))
            
    
if __name__ == '__main__':
    main()
    

Command line:

Code: Select all

py -3 remove_iden_epd.py -i sample.epd -o out.epd

Sample.epd

Code: Select all

8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - bm Bh5+; id "W.Eigenmann: Brillant 002";
8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - id ?; bm Bh5;
8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id "arasan19.45"; c0 "Camacho Martinez-An. C. Hernandez, Cuba 1995";
8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id peg124;

out.epd

Code: Select all

8/2N4r/1p3pkp/8/5K1p/2P4N/P3Bn2/8 w - - bm Bh5+; id "W.Eigenmann: Brillant 002";
8/2p1k3/3p3p/2PP1pp1/1P1K1P2/6P1/8/8 w - - bm g4; id "arasan19.45"; c0 "Camacho Martinez-An. C. Hernandez, Cuba 1995";

[Spill_The_Tea]

Wow That was fast.
I did not expect that. Thanks Ferdy!
I'll test it out, and see how it works.

[Dann Corbit]

I have a database that I store EPD records into.
I parse the EPD first, using the grammar of EPD records.
If the main EPD (Position + Castle rights + E.P.) has never been seen before, I simply insert it, along with all other EPD fields present.
If I have seen the EPD before, I only replace null fields with new data.
The only exception is that if analysis is deeper, I take the deepest one, and move the shallow data into AlternateEvals.
Otherwise, I stick the new analysis into a table called "AlternateEvals".
That is for my main EPD table.

But I also have individual tables for various contests, and I follow similar rules for these contests.
For all the contest tables, I collect the engine name from the PGN header, and also the result and save those as well for the position.
I keep a relation table, which has the engine name along with Elo and a few other facts.

But this problem is a very difficult one.
I wish I had, from the start, stored the original engine and hardware in a relation table, but I neglected to do that for my main table (EPD).

[Spill_The_Tea]

I have a database that I store EPD records into.
I parse the EPD first, using the grammar of EPD records.
If the main EPD (Position + Castle rights + E.P.) has never been seen before, I simply insert it, along with all other EPD fields present.
If I have seen the EPD before, I only replace null fields with new data.
The only exception is that if analysis is deeper, I take the deepest one, and move the shallow data into AlternateEvals.
Otherwise, I stick the new analysis into a table called "AlternateEvals".
That is for my main EPD table.

But I also have individual tables for various contests, and I follow similar rules for these contests.
For all the contest tables, I collect the engine name from the PGN header, and also the result and save those as well for the position.
I keep a relation table, which has the engine name along with Elo and a few other facts.

But this problem is a very difficult one.
I wish I had, from the start, stored the original engine and hardware in a relation table, but I neglected to do that for my main table (EPD).

Hi Dann,
If I remember correctly, you use Arena (or an alternative GUI) to run your epd-test suites? I ask because I typically use the polyglot epd-test functionality with some modifications, or if I am searching for difficult positions, this python script from poorfish github.

But if the output of your epd test suite is a pgn file, I use this python script from Giraffe, to convert pgn fen tags to epd files.


1. That said, Do you do manage the results of your database manually, inserting each new result into a spreadsheet? or do you have a script for this?
a. If not, it sounds like you need a script to parse the fen, Find Identical fen in results database, and compare acd op code between both results. If new result depth > old depth, rewrite acd, acs, acn, and pv; else, print to alternative spreadsheet?

2. Opinion Piece: Standardized op codes for EPD files is rather Limiting/generic; In contrast, pgn header tags are rather accommodating to additional arguments as needed (beyond the standard 7). As long as you throw it between two brackets, it is valid. I would seriously consider adding op codes as per your needs, such as Engine (eng), Version (ver), Threads/Cores (thr), Hash (mb).

[Spill_The_Tea]

Hi Ferdy,
I tested the script, and ran into an issue with utf-8 encoding, so I altered the code, importing codecs in order to ignore utf-8 related errors:

Code: Select all

import argparse
import codecs

VERSION = 'v0.1.0'


def main():
    parser = argparse.ArgumentParser(prog='Identical epd remover {}'.format(VERSION))
    parser.add_argument('-i', '--input', help='input epd filename',
                        required=True)
    parser.add_argument('-o', '--output', help='output filename, overwrite mode',
                                            required=True)
                        
    args = parser.parse_args()
    epdfn = args.input
    outfn = args.output
                        
    tmp_save = []
    uni_save = []
                        
    # Read input epd and remove identical, consider first entry as priority
    with codecs.open(epdfn, 'r', encoding='utf-8',
                     errors='ignore') as f:
        for lines in f:
            line = lines.strip()
                            
            base_epd = ' '.join(line.split()[0:4]).strip()
            if base_epd not in tmp_save:
                tmp_save.append(base_epd)
                uni_save.append(line)

    # Write to output file
    with open(outfn, 'w') as f:
        for epd_line in uni_save:
            f.write('{}\n'.format(epd_line))


if __name__ == '__main__':
    main()
                 

[Ferdy]

Hi Ferdy,
I tested the script, and ran into an issue with utf-8 encoding, so I altered the code, importing codecs in order to ignore utf-8 related errors:

Code: Select all

import argparse
import codecs

VERSION = 'v0.1.0'


def main():
    parser = argparse.ArgumentParser(prog='Identical epd remover {}'.format(VERSION))
    parser.add_argument('-i', '--input', help='input epd filename',
                        required=True)
    parser.add_argument('-o', '--output', help='output filename, overwrite mode',
                                            required=True)
                        
    args = parser.parse_args()
    epdfn = args.input
    outfn = args.output
                        
    tmp_save = []
    uni_save = []
                        
    # Read input epd and remove identical, consider first entry as priority
    with codecs.open(epdfn, 'r', encoding='utf-8',
                     errors='ignore') as f:
        for lines in f:
            line = lines.strip()
                            
            base_epd = ' '.join(line.split()[0:4]).strip()
            if base_epd not in tmp_save:
                tmp_save.append(base_epd)
                uni_save.append(line)

    # Write to output file
    with open(outfn, 'w') as f:
        for epd_line in uni_save:
            f.write('{}\n'.format(epd_line))


if __name__ == '__main__':
    main()
                 

Without using import codecs, try this if it will work.
Just add the parameter encoding='utf-8' in open(), twice both in reading and writing.

Code: Select all

import argparse

VERSION = 'v0.1.0'


def main():
    parser = argparse.ArgumentParser(prog='Identical epd remover {}'.format(VERSION))
    parser.add_argument('-i', '--input', help='input epd filename',
                        required=True)
    parser.add_argument('-o', '--output', help='output filename, overwrite mode',
                                            required=True)
                        
    args = parser.parse_args()
    epdfn = args.input
    outfn = args.output
                        
    tmp_save = []
    uni_save = []
                        
    # Read input epd and remove identical, consider first entry as priority
    with open(epdfn, encoding='utf-8') as f:
        for lines in f:
            line = lines.strip()
                            
            base_epd = ' '.join(line.split()[0:4]).strip()
            if base_epd not in tmp_save:
                tmp_save.append(base_epd)
                uni_save.append(line)

    # Write to output file
    with open(outfn, 'w', encoding='utf-8') as f:
        for epd_line in uni_save:
            f.write('{}\n'.format(epd_line))


if __name__ == '__main__':
    main()
           

[Dann Corbit]

I have a database that I store EPD records into.
I parse the EPD first, using the grammar of EPD records.
If the main EPD (Position + Castle rights + E.P.) has never been seen before, I simply insert it, along with all other EPD fields present.
If I have seen the EPD before, I only replace null fields with new data.
The only exception is that if analysis is deeper, I take the deepest one, and move the shallow data into AlternateEvals.
Otherwise, I stick the new analysis into a table called "AlternateEvals".
That is for my main EPD table.

But I also have individual tables for various contests, and I follow similar rules for these contests.
For all the contest tables, I collect the engine name from the PGN header, and also the result and save those as well for the position.
I keep a relation table, which has the engine name along with Elo and a few other facts.

But this problem is a very difficult one.
I wish I had, from the start, stored the original engine and hardware in a relation table, but I neglected to do that for my main table (EPD).

Hi Dann,
If I remember correctly, you use Arena (or an alternative GUI) to run your epd-test suites? I ask because I typically use the polyglot epd-test functionality with some modifications, or if I am searching for difficult positions, this python script from poorfish github.

But if the output of your epd test suite is a pgn file, I use this python script from Giraffe, to convert pgn fen tags to epd files.


1. That said, Do you do manage the results of your database manually, inserting each new result into a spreadsheet? or do you have a script for this?
a. If not, it sounds like you need a script to parse the fen, Find Identical fen in results database, and compare acd op code between both results. If new result depth > old depth, rewrite acd, acs, acn, and pv; else, print to alternative spreadsheet?

Everything is automated.
Most of my analysis uses open source programs like Stockfish that I have modified to write logging directly to disk. I use Arena sometimes when I am running an engine for which I do not have the source code (typically Komodo or Houdini). I have a program that stitches together the arena log and the input files to create decorated EPD records when I do it that way. It's not very good because Arena will become disconnected from the engine form time to time and start writing analysis to the wrong record slot in the output. So I always pipe that output to a program that I have that detects illegal operations, and if I detect any, I manually examine the output. So I don't like to use Arena for analysis unless I really have to for some reason.
When I get an analyzed PGN file, I use a tool I got from my friend Les Fernandez that transforms the PGN files into EPD records that contain the analysis and the game statistics.
When I insert the data into my SQL database, I have programs I have written that read the PGN files and write stored procedure calls (the inserts and/or merge operations are very complicated).
So I just schedule processes and pipe the output from one program to the next.
There is very little work on my part.

2. Opinion Piece: Standardized op codes for EPD files is rather Limiting/generic; In contrast, pgn header tags are rather accommodating to additional arguments as needed (beyond the standard 7). As long as you throw it between two brackets, it is valid. I would seriously consider adding op codes as per your needs, such as Engine (eng), Version (ver), Threads/Cores (thr), Hash (mb).

I don't care about threads or hash. I do care about depth and engine version. So I do not record hash, threads, or even tablebase data from analysis. But I do collect engine information, depth, time. I collect node counts, but those are just historical artifacts. I do not use them anymore for anything and so I do not worry about getting the multipliers (K,M,B) and that sort of thing.

[Ferdy]

MEA (Multiple-move-Epd-Analyzer) in github at https://github.com/fsmosca/Multiple-move-Epd-Analyzer

No exe yet, will upload later. Last uploaded exe (was older) was in
https://drive.google.com/file/d/1SUte6B ... sp=sharing, this was version 0.3.5, current version is 0.3.7 but have no major changes since v0.3.5.