Looking for a (special) EPD to PGN util

[Rebel]

[d]r1bk1n1r/pp1n1q1p/2p2p1R/3p4/3PpN2/2NB2Q1/PPP2PP1/2K1R3 w - -
[pgn][Event "?"]
[Site "?"]
[Date "????.??.??"]
[White "NN"]
[Black "NN"]
[Result "*"]

1.e4 e6 2.d4 d5 3.Nc3 Nf6 4.Bg5 Be7 5.e5 Nfd7 6.h4 Bxg5 7.hxg5 Qxg5 8.Nh3
Qe7 9.Nf4 Nf8 10.Qg4 f5 11.exf6 gxf6 12.O-O-O c6 13.Re1 Kd8 14.Rh6 e5 15.
Qh4 Nbd7 16.Bd3 e4 17.Qg3 Qf7 *[/pgn]
I am looking for a util that from an EPD creates a PGN with move history as in the above example.

Does it exist?

If not then:
1. Do I hear anyone volunteer?
2. or must I write it myself once again?

Kidding aside, it's handy for playing eng-eng matches, one negative aspect of playing from EPD positions is that the engine has no spare time as it does when you play from predefined openings from PGN.

[Dann Corbit]

The problem is that there are an enormous number of move sequences that lead to that position.
I guess you want the most frequent move sequence that leads to that pattern.
When you do an opening report with SCID, it gives the most frequent move sequence to the given position.
But I do not think you want to run a SCID opening report for every position in a large test suite, especially if it is enormous like MEA or NICE.

It might be possible to call scid routines and accomplish it, but I guess it would be a lot of work.

[Dann Corbit]

I have an idea.
You could take a tool like pgn2fen and feed it a titanic database of PGN games, producing a megatitanic set of EPD.
Now, search for a given position in the megatitanic test set. You can then go backwards until you find the opening position and you will have found "A" sequence that leads to the given move.
Of course, there is no sure way to know if the position will be in your database to start with.

Another thing to look at is existing "retrograde analysis" chess tools. I think they might do what you want.
Maybe:
http://xenon.stanford.edu/~hwatheod/Retractor/
or with natch:
Natch solves retrograde analysis problems by constructing "proof games" - the shortest possible game leading to a certain position. Natch is a command line utility, but there is a Java based graphical interface, iNatch (110 kb). To run iNatch you must have SUN's Java Runtime Environment 1.3 (or higher) installed.
Link: The Natch homepage: http://natch.free.fr/Natch.html

But this makes it sound like a very difficult problem:
https://webdocs.cs.ualberta.ca/~jonatha ... abases.pdf

[hgm]

You could try it with a 'guided perft': prune all moves that move a Pawn beyond the square where your target position has a Pawn, and all moves with a piece that is already on the target position. And prune moves of a piece that do not put it in the target location when it does have a move that could put it on the target location. My guess is that for opening positions that should get you there rather quickly. There will be very many transpositions, but you only want to find one, and can abort the perft when you have it.

[Vinvin]

...
I am looking for a util that from an EPD creates a PGN with move history as in the above example.

Does it exist?

Yes, that's a problem of "retrograde analysis" https://en.wikipedia.org/wiki/Proof_game .
Sometimes it's impossible to find a way to reach a position.
Natch is good to construct a proof game : http://natch.free.fr/Natch.html

Code: Select all

This file contains an example of a position in Forsythe notation and
explanations about what is allowed and what is not in such files. It
may be used without any modification by Natch.

Here is the position for human eyes :

Pascal WASSONG
Phenix 17, june 1992
+---+---+---+---+---+---+---+---+
|   |*R |*B |*Q |   | . |*S |*R |
+---+---+---+---+---+---+---+---+
|*P |   | . |   | . |*P |*P |*P |
+---+---+---+---+---+---+---+---+
|   |*B |*S |*P |   | . |   | . |
+---+---+---+---+---+---+---+---+
| . |*P |*P |   |*P |   | . |   |
+---+---+---+---+---+---+---+---+
|*K | . |   | . |   | . |   | . |
+---+---+---+---+---+---+---+---+
| . |   | . |   | . | P | . |   |
+---+---+---+---+---+---+---+---+
| P | P | P | P | P | . | P | P |
+---+---+---+---+---+---+---+---+
| R | S | B | Q | K | B | S | R |
+---+---+---+---+---+---+---+---+
16 + 16                  SPG 12.0

All the lines of this file are ignored except the 2 following this
paragraph. They describe the position which is solved by Natch. The
second line must begin with the number of single moves of the
solution. The rest of the second line is also ignored.

1rbq2sr/p4ppp/1bsp4/1pp1p3/k7/5P2/PPPPP1PP/RSBQKBSR
24 single moves

In the english notation, the characters `s' and `n' are equivalent for
the knight.

Natch searches for the first line containing only authorized symbols
and exactly 2 kings. For each line Natch tries out the 3
languages. The language of the position is not used to set the
language in which Natch writes the solution. You have to use the `-l'
option to change the output.

The same position in french notation ...
1tfd2ct/p4ppp/1fcp4/1pp1p3/r7/5P2/PPPPP1PP/TCFDRFCT
24 demi-coups

... and in german.
1tld2st/b4bbb/1lsb4/1bb1b3/k7/5B2/BBBBB1BB/TSLDKLST
24 Halbzuegen

Natch will only solve the first position found in the file, which in
this case is the english one. If you want to test the other lines, put
for example an exclamation point (`!') at the beginning of the line
containing the english position.

The solution of this problem is :
 1.Pf2-f3   Pe7-e5    2.Ke1-f2   Bf8-c5    3.Kf2-g3   Pd7-d6   
 4.Qd1-e1   Ke8-d7    5.Qe1-f2   Kd7-c6    6.Qf2-d4   Kc6-b5   
 7.Qd4-e3   Kb5-a4    8.Qe3-f2   Pb7-b5    9.Qf2-e1   Bc5-b6   
10.Qe1-d1   Pc7-c5   11.Kg3-f2   Sb8-c6   12.Kf2-e1   Ra8-b8   

[Vinvin]

For the position you posted :

Code: Select all

Solution 1 (position 1)
1.Sg1-h3   Sg8-f6   2.Sb1-c3   Pd7-d5   3.Pd2-d4   Sf6-d7   
4.Bc1-f4   Pc7-c6   5.Bf4-d6   Qd8-c7   6.Pe2-e4   Pd5xe4   
7.Qd1-g4   Pe7xd6   8.Qg4xg7   Pd6-d5   9.O-O-O    Bf8-d6   
10.Rd1-e1   Bd6xh2   11.Qg7-g3   Sd7-f8   12.Sh3-f4   Ke8-d8   
13.Rh1xh2   Pf7-f6   14.Rh2-h6   Qc7-f7   15.Bf1-d3   Sb8-d7   

[Fulvio]

It might be possible to call scid routines and accomplish it, but I guess it would be a lot of work.

Save this script to a file mysearch.tcl

Code: Select all

lassign $argv fnameSrc fnameRef
set srcdb [sc_base open PGN $fnameSrc]
set refdb [sc_base open PGN $fnameRef]
set outdb [sc_base create PGN "output.pgn"]

set ngames [sc_base numGames $srcdb]
for {set i 1} {$i <= $ngames} {incr i} {
	sc_base switch $srcdb
	sc_game load $i
	set comment [sc_pos getComment]

	sc_filter search $refdb tree board
	lassign [sc_base gameslist $refdb 0 1 tree N+] firstMatch
	if {$firstMatch == ""} {
		# Not found, save to output.pgn the original FEN
		sc_game save 0 $outdb
		continue
	}
	
	lassign [split $firstMatch _] gnum ply
	sc_base switch $refdb
	sc_game load $gnum
	sc_move ply $ply
	sc_game truncate
	sc_pos setComment $comment
	sc_game save 0 $outdb
}

Search the positions from temere.epd in refdb.pgn and create an output.pgn:

Code: Select all

scid.exe mysearch.tcl temere.epd refdb.pgn

There is no error handling and on windows the command returns immediately.
I used the task manager to wait for it to finish (about 5 minutes). In a 2M games database it found 1321 positions out of 4975.
Hope it helps.

[Rebel]

It might be possible to call scid routines and accomplish it, but I guess it would be a lot of work.

Save this script to a file mysearch.tcl

Code: Select all

lassign $argv fnameSrc fnameRef
set srcdb [sc_base open PGN $fnameSrc]
set refdb [sc_base open PGN $fnameRef]
set outdb [sc_base create PGN "output.pgn"]

set ngames [sc_base numGames $srcdb]
for {set i 1} {$i <= $ngames} {incr i} {
	sc_base switch $srcdb
	sc_game load $i
	set comment [sc_pos getComment]

	sc_filter search $refdb tree board
	lassign [sc_base gameslist $refdb 0 1 tree N+] firstMatch
	if {$firstMatch == ""} {
		# Not found, save to output.pgn the original FEN
		sc_game save 0 $outdb
		continue
	}
	
	lassign [split $firstMatch _] gnum ply
	sc_base switch $refdb
	sc_game load $gnum
	sc_move ply $ply
	sc_game truncate
	sc_pos setComment $comment
	sc_game save 0 $outdb
}

Search the positions from temere.epd in refdb.pgn and create an output.pgn:

Code: Select all

scid.exe mysearch.tcl temere.epd refdb.pgn

There is no error handling and on windows the command returns immediately.
I used the task manager to wait for it to finish (about 5 minutes). In a 2M games database it found 1321 positions out of 4975.
Hope it helps.

That's impressive, can you post the first 10 raw PGN's ?

[Fulvio]

That's impressive, can you post the first 10 raw PGN's ?

Here it is:

Code: Select all

[Event "18th WChCorr"]
[Site "ICCF COR"]
[Date "2004.06.01"]
[Round "1"]
[White "Mraz, M."]
[Black "Umansky, M."]
[Result "1/2-1/2"]
[WhiteElo "2559"]
[BlackElo "2654"]

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.e3 c5 5.Bd3 Nc6 6.Ne2 cxd4 7.exd4 d5 8.cxd5 
Nxd5 9.O-O O-O 10.Nxd5 Qxd5 11.a3 Bd6 12.Nc3 Qa5 13.Be3 f5 14.f4 Bd7 15.
Qf3 Rae8 16.Rfe1 Kh8 17.Bc4 Qd8 18.Rac1 b6 19.Bf2 Qf6 20.g3 Bb8 21.d5 Na5 
{bm dxe6; c0 "dxe6=10, Bb5=8, Ba6=0, Bf1=0";} 1/2-1/2

[Event "18th WChCorr"]
[Site "ICCF COR"]
[Date "2004.06.01"]
[Round "1"]
[White "Mraz, M."]
[Black "Umansky, M."]
[Result "1/2-1/2"]
[WhiteElo "2559"]
[BlackElo "2654"]

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.e3 c5 5.Bd3 Nc6 6.Ne2 cxd4 7.exd4 d5 8.cxd5 
Nxd5 9.O-O O-O 10.Nxd5 Qxd5 11.a3 Bd6 12.Nc3 Qa5 13.Be3 f5 14.f4 Bd7 15.
Qf3 Rae8 16.Rfe1 Kh8 17.Bc4 Qd8 18.Rac1 b6 19.Bf2 Qf6 20.g3 Bb8 21.d5 Na5 
22.dxe6 Bxe6 23.Rxe6 Rxe6 24.Bxe6 Qxe6 25.Nb5 {bm Nb3; c0 "Nb3=10, Qd7=0, 
Qb3=0, Qf7=0";} 1/2-1/2

[Event "18th WChCorr"]
[Site "ICCF COR"]
[Date "2004.06.01"]
[Round "1"]
[White "Mraz, M."]
[Black "Umansky, M."]
[Result "1/2-1/2"]
[WhiteElo "2559"]
[BlackElo "2654"]

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.e3 c5 5.Bd3 Nc6 6.Ne2 cxd4 7.exd4 d5 8.cxd5 
Nxd5 9.O-O O-O 10.Nxd5 Qxd5 11.a3 Bd6 12.Nc3 Qa5 13.Be3 f5 14.f4 Bd7 15.
Qf3 Rae8 16.Rfe1 Kh8 17.Bc4 Qd8 18.Rac1 b6 19.Bf2 Qf6 20.g3 Bb8 21.d5 Na5 
22.dxe6 Bxe6 23.Rxe6 Rxe6 24.Bxe6 Qxe6 25.Nb5 Nb3 26.Rd1 Nc5 27.Qd5 Qc8 
28.Bxc5 bxc5 29.Rc1 a6 {bm Nd4; c0 "Nd4=10, Nc3=0, Nd6=0, Rxc5=0";} 
1/2-1/2

[Event "18th WChCorr"]
[Site "ICCF COR"]
[Date "2004.06.01"]
[Round "1"]
[White "Mraz, M."]
[Black "Umansky, M."]
[Result "1/2-1/2"]
[WhiteElo "2559"]
[BlackElo "2654"]

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.e3 c5 5.Bd3 Nc6 6.Ne2 cxd4 7.exd4 d5 8.cxd5 
Nxd5 9.O-O O-O 10.Nxd5 Qxd5 11.a3 Bd6 12.Nc3 Qa5 13.Be3 f5 14.f4 Bd7 15.
Qf3 Rae8 16.Rfe1 Kh8 17.Bc4 Qd8 18.Rac1 b6 19.Bf2 Qf6 20.g3 Bb8 21.d5 Na5 
22.dxe6 Bxe6 23.Rxe6 Rxe6 24.Bxe6 Qxe6 25.Nb5 Nb3 26.Rd1 Nc5 27.Qd5 Qc8 
28.Bxc5 bxc5 29.Rc1 a6 30.Nd4 {bm Ba7; c0 "Ba7=10, h6=4, Qe8=0, a5=0";} 
1/2-1/2

[Event "Germany"]
[Site "Germany"]
[Date "1989.01.01"]
[Round "1"]
[White "Weiss_Nowak, Christ"]
[Black "Blauert, Jorg"]
[Result "0-1"]

1.d4 Nf6 2.c4 e6 3.Nc3 d5 4.cxd5 exd5 5.Bg5 Be7 6.e3 c6 7.Qc2 Nbd7 8.Bd3 
Nh5 9.Bxe7 Qxe7 10.O-O-O g6 11.Nf3 Nb6 12.h3 Be6 13.Kb1 O-O-O 14.Na4 Nxa4 
15.Qxa4 Kb8 {bm Rc1; c0 "Rc1=10, Qc2=0, Nd2=0, Rd2=0";} 0-1

[Event "Smartfish Masters"]
[Site "Drammen NOR"]
[Date "2005.01.04"]
[Round "1"]
[White "Khalifman, A."]
[Black "Johannessen, LE."]
[Result "1/2-1/2"]
[WhiteElo "2669"]
[BlackElo "2519"]

1.e4 c6 2.d4 d5 3.Nc3 dxe4 4.Nxe4 Bf5 5.Ng3 Bg6 6.h4 h6 7.Nf3 Nd7 8.h5 Bh7
9.Bd3 Bxd3 10.Qxd3 e6 11.Bf4 Qa5+ 12.Bd2 Qc7 13.O-O-O O-O-O 14.Ne4 Ngf6 
15.g3 Nxe4 16.Qxe4 Bd6 17.c4 c5 18.d5 Nf6 19.Qc2 exd5 20.cxd5 Qd7 21.Bc3 
Rhe8 22.Bxf6 gxf6 23.Kb1 Qg4 24.Nd4 Re5 25.Nb5 Kb8 26.f4 Rxh5 {bm Qa4; c0 
"Qa4=10, Rhe1=7, Rxh5=7, Nxd6=1";} 1/2-1/2

[Event "18th WChCorr"]
[Site "ICCF COR"]
[Date "2004.06.01"]
[Round "1"]
[White "Spitz, P."]
[Black "Toro Solis de Ovand"]
[Result "1/2-1/2"]
[WhiteElo "2548"]
[BlackElo "2636"]

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Bg5 e6 7.f4 Qb6 8.Nb3 Be7
9.Be2 Nbd7 10.Qd2 h6 11.Bxf6 Nxf6 12.O-O-O Qc7 13.Bf3 Bd7 14.Kb1 O-O-O 15.
Rhe1 Kb8 16.Nd4 Be8 17.Re3 {bm g5; c0 "Nd7=10, g5=9, h5=8, g6=8";} 1/2-1/2

[Event "18th WChCorr"]
[Site "ICCF COR"]
[Date "2004.06.01"]
[Round "1"]
[White "Spitz, P."]
[Black "Toro Solis de Ovand"]
[Result "1/2-1/2"]
[WhiteElo "2548"]
[BlackElo "2636"]

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Bg5 e6 7.f4 Qb6 8.Nb3 Be7
9.Be2 Nbd7 10.Qd2 h6 11.Bxf6 Nxf6 12.O-O-O Qc7 13.Bf3 Bd7 14.Kb1 O-O-O 15.
Rhe1 Kb8 16.Nd4 Be8 {bm g3; c0 "g3=10, h4=6, a3=4, Re3=1";} 1/2-1/2

[Event "FIDE (37) 1990-1993"]
[Site "El Escorial cm f"]
[Date "1993.01.01"]
[Round "1"]
[White "Timman, J."]
[Black "Short, N."]
[Result "1-0"]
[WhiteElo "2635"]
[BlackElo "2655"]

1.d4 d5 2.c4 e6 3.Nc3 Nf6 4.cxd5 exd5 5.Bg5 c6 6.Qc2 Be7 7.e3 Nbd7 8.Bd3 
Nh5 9.Bxe7 Qxe7 10.Nge2 Nb6 11.O-O-O g6 12.Kb1 Ng7 13.Ng3 Bd7 14.Rc1 O-O-O
15.Na4 Nxa4 16.Qxa4 Kb8 17.Rc3 b6 18.Ra3 Be8 19.Qc2 Rc8 {bm Rc1; c0 
"Qd1=10, Qe2=7, Rc1=10, Ne2=9";} 1-0

[Event "Cappelle la Grande"]
[Site "Cappelle la Grande"]
[Date "1991.01.01"]
[Round "1"]
[White "Anastasian, A."]
[Black "Conquest, S."]
[Result "1/2-1/2"]

1.d4 Nf6 2.Nf3 e6 3.c4 c5 4.d5 b5 5.dxe6 fxe6 6.cxb5 Bb7 7.g3 Be7 8.Bg2 a6
9.bxa6 Rxa6 10.O-O O-O 11.Nbd2 {bm d5; c0 "Nc6=10, d5=9, Nc6=5, Qe8=3";} 
1/2-1/2

The longest game matched at move 78:

Code: Select all

[Event "10th Agzamov Mem 2016"]
[Site "Tashkent UZB"]
[Date "2016.04.20"]
[Round "1"]
[White "Amonatov, F"]
[Black "Egin, V"]
[Result "1-0"]
[WhiteElo "2602"]
[BlackElo "2348"]

1.e4 e5 2.Nf3 Nf6 3.d4 Nxe4 4.Bd3 d5 5.dxe5 Be7 6.O-O Nc5 7.Be2 O-O 8.Be3 
c6 9.c4 dxc4 10.Bxc4 Qxd1 11.Rxd1 Nbd7 12.Nc3 Nb6 13.Be2 Bg4 14.h3 Bxf3 
15.gxf3 Rad8 16.f4 f5 17.b4 Ne6 18.Bxb6 axb6 19.Bc4 Kf7 20.Ne2 b5 21.Bb3 
c5 22.bxc5 Bxc5 23.Nc3 Bd4 24.Rac1 Bxc3 25.Rxd8 Rxd8 26.Rxc3 Ke7 27.Bxe6 
Kxe6 28.Rc7 Rd4 29.Rxg7 Rxf4 30.Rxh7 Kxe5 31.Rxb7 Rb4 32.Kg2 Kf4 33.Rb6 
Ra4 34.Rxb5 Rxa2 35.Rb4+ Kg5 36.Kg3 Ra3+ 37.f3 Ra5 38.h4+ Kh5 39.Rb8 Kg6 
40.Rb6+ Kg7 41.Kf4 Ra3 42.Rc6 Rb3 43.Re6 Kf7 44.Re3 Rb6 45.Kxf5 Rf6+ 46.
Kg4 Rg6+ 47.Kh3 Rg1 48.Re4 Ra1 49.Kg4 Kf6 50.Rb4 Ra6 51.h5 Kg7 52.f4 Kh6 
53.Re4 Ra1 54.Re6+ Kg7 55.Kg5 Rg1+ 56.Kf5 Rh1 57.h6+ Kh7 58.Kg5 Rg1+ 59.
Kf5 Rh1 60.Ke5 Re1+ 61.Kf6 Ra1 62.Kf7 Ra7+ 63.Re7 Ra5 64.Re6 Ra7+ 65.Kf8 
Ra8+ 66.Ke7 Ra7+ 67.Kd6 Ra6+ 68.Ke5 Ra5+ 69.Ke4 Ra4+ 70.Kf5 Ra1 71.Rd6 Rh1
72.Re6 Rf1 73.Kg5 Rg1+ 74.Kf5 Rf1 75.Ke5 Re1+ 76.Kf6 Ra1 77.f5 Kxh6 78.
Kf7+ 
    {bm Kh7; c0 "Kh7=10, Kg5=0, Kh5=0";}
1-0