how to create a labeled epd from pgn?

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
zenpawn
Posts: 262
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

how to create a labeled epd from pgn?

Post by zenpawn » Sat Dec 02, 2017 2:33 am

I'd like to create my own set of positions with game result labels for Texel tuning, ala the quiet-labeled.epd file shared on this forum, which contains lines such as:

rn2kb1r/ppp1pp1p/2q3p1/3nN3/3P4/2N1P3/PPb2PPP/R1B1KB1R b KQkq - c9 "0-1";

The first two steps in the process are easy enough: 1) generate a file of quiet position FENs, 2) run fast games starting from these positions to get results. Is there a script that will take the PGN from #2 and create a labeled epd file combining the starting positions with their results (either in a comment or just appended to the end of the line)?

Thanks,
-Erin

AlvaroBegue
Posts: 884
Joined: Tue Mar 09, 2010 2:46 pm
Location: New York

Re: how to create a labeled epd from pgn?

Post by AlvaroBegue » Sat Dec 02, 2017 6:49 am

zenpawn wrote:I'd like to create my own set of positions with game result labels for Texel tuning, ala the quiet-labeled.epd file shared on this forum, which contains lines such as:

rn2kb1r/ppp1pp1p/2q3p1/3nN3/3P4/2N1P3/PPb2PPP/R1B1KB1R b KQkq - c9 "0-1";

The first two steps in the process are easy enough: 1) generate a file of quiet position FENs, 2) run fast games starting from these positions to get results. Is there a script that will take the PGN from #2 and create a labeled epd file combining the starting positions with their results (either in a comment or just appended to the end of the line)?

Thanks,
-Erin
Since the FEN and the result are both described in the header of the PGN file, it should be very easy to parse. I would use Perl to do it. If you post an example game from your database, I'll give you a tiny piece of Perl that will do what you want.

zenpawn
Posts: 262
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: how to create a labeled epd from pgn?

Post by zenpawn » Sat Dec 02, 2017 10:32 am

That would be great. Here's an example:

Code: Select all

[Event "SF8-SF8-STS-10sec"]
[Site "ROOK"]
[Date "2017.12.02"]
[Round "21"]
[White "Stockfish_8_x64"]
[Black "Stockfish_8_x64 [001]"]
[Result "1/2-1/2"]
[BlackElo "3456"]
[Time "05:21:30"]
[WhiteElo "3456"]
[TimeControl "10+0"]
[SetUp "1"]
[FEN "1rb3k1/2pn2pp/p2p4/4p3/1pP4q/1P1PBP1P/1PQ2P2/R3R1K1 w - - 0 1"]
[Termination "normal"]
[PlyCount "14"]
[WhiteType "program"]
[BlackType "program"]

1. c5 {(c4-c5 d6-d5 Be3-c1 c7-c6 d3-d4 Qh4xh3 d4xe5 Qh3xf3 e5-e6 Qf3-g4+
Kg1-f1 Qg4-h3+ Kf1-g1 Qh3-g4+) 0.00/14 0} d5 {(d6-d5 Be3-c1 c7-c6 d3-d4
Qh4xh3 Qc2-e2 Qh3-h5 Qe2-e3 h7-h6 Bc1-d2 e5xd4 Qe3xd4 Nd7xc5 Qd4xc5 Qh5-g6+
Kg1-f1 Qg6-d3+ Kf1-g1 Qd3xd2 Re1-e8+ Kg8-h7 Qc5-f8 Qd2-g5+ Kg1-h1 Qg5-h5+
Kh1-g1 Qh5-g5+) 0.00/16 0} 2. Bc1 {(Be3-c1 c7-c6 d3-d4 Qh4xd4 Bc1-g5 Qd4xc5
Qc2xc5 Nd7xc5 Re1xe5 h7-h6 Bg5-e7 Nc5xb3 Ra1-d1 Kg8-f7 Be7-d6 Rb8-b7 Kg1-h2
Nb3-a5 Rd1-g1 b4-b3 Rg1-g3) +0.22/15 0} Qf6 {(Qh4-f6 Qc2-e2 Nd7xc5 Qe2xe5
Qf6xe5 Re1xe5 Nc5xb3 Ra1-b1 Bc8xh3 Kg1-h2 Bh3-d7 Re5-e7 Bd7-b5 Bc1-e3 d5-d4
Rb1-g1 g7-g6 Be3-f4 Rb8-f8 Bf4xc7 Bb5xd3) +0.12/17 1} 3. f4 {(f3-f4 Qf6-g6+
Kg1-h2 Qg6-f5 Kh2-g1 Qf5-g6+) 0.00/17 1} Qg6+ {(Qf6-g6+ Kg1-h2 Qg6-f5
Kh2-g1 Qf5-g6+) 0.00/18 0} 4. Kh2 {(Kg1-h2 Qg6-f5 Kh2-g1 Qf5-g6+) 0.00/19
0} Qf5 {(Qg6-f5 Kh2-g1 Qf5-g6+) 0.00/19 0} 5. Kg1 {(Kh2-g1 Qf5-g6+) 0.00/19
0} Qg6+ {(Qf5-g6+) 0.00/19 0} 6. Kh2 {(Kg1-h2) 0.00/26 0} Qf5 {(Qg6-f5)
0.00/23 0} 7. Kg1 {(Kh2-g1) 0.00/23 0} Qg6+ {(Qf5-g6+) 0.00/19 0 3-fold
repetition} 1/2-1/2

brtzsnr
Posts: 426
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: how to create a labeled epd from pgn?

Post by brtzsnr » Sat Dec 02, 2017 11:12 am

Given a set of PGNs I use the following command (works on my Debian)

Code: Select all

cat ok/pgn-split.* | grep "FEN\|Result\|ECO" | tac | paste - - -d' ' | grep -v ECO | sed 's/\[FEN "//' | sed 's/0 1"\] \[Result/c9/' | sed 's/\]/;/' > quiet-labeled.epd

Let's split:

Code: Select all

cat ok/pgn-split.*
just concatenates everything

Code: Select all

grep "FEN\|Result\|ECO"
keeps only the lines containing FEN, ECO or RESULT

Code: Select all

tac
reverses the lines

Code: Select all

paste - - -d' ' 
merges the lines two by two

Code: Select all

grep -v ECO
removes ECO lines. Some cutechess versions output ECO instead of FEN

Code: Select all

sed 's/\[FEN "//' | sed 's/0 1"\] \[Result/c9/' | sed 's/\]/;/'
removes extra characters and adds c9.

jdart
Posts: 3507
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: how to create a labeled epd from pgn?

Post by jdart » Sat Dec 02, 2017 2:20 pm

I have two tools for this in my source tree (https://github.com/jdart1/arasan-chess):

1. A program called pgnselect that takes a PGN file and produces a sampled set of EPD positions (unlabeled). If you run it with the -q switch, it will select quiet positions. This is built from my Makefile using "make utils".

2. A python3 script named "label_positions.py" in the tools directory that takes the unlabeled EPD positions and add labels, using cutechess-cli. Cutechess-cli should be set up with an engines.json file containing the engine configuration. This script will probably require a few edits to adapt to your environment.

--Jon

zenpawn
Posts: 262
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: how to create a labeled epd from pgn?

Post by zenpawn » Sat Dec 02, 2017 2:37 pm

Thanks, everyone. These ought to do it. :)

AlvaroBegue
Posts: 884
Joined: Tue Mar 09, 2010 2:46 pm
Location: New York

Re: how to create a labeled epd from pgn?

Post by AlvaroBegue » Sat Dec 02, 2017 3:08 pm

Code: Select all

cat something.pgn | perl -ne '$result = $1 if /Result (\".*\")/; print "$1 $result\n" if /FEN \"(.*)\"/'
The invocation `perl -ane' means "for each line of input, execute the following piece of Perl code". What the code does is remember the result if it finds a line that matches some pattern, and print out a line if it matches a different pattern. `$1' is the part of the pattern captured between the parentheses.

zenpawn
Posts: 262
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: how to create a labeled epd from pgn?

Post by zenpawn » Sat Dec 02, 2017 3:18 pm

Good stuff. Thanks.

Post Reply