Tool for converting PGN to EPD for analysis.

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

Dann Corbit
Posts: 12792
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Tool for converting PGN to EPD for analysis.

Post by Dann Corbit »

Ferdy wrote:
reflectionofpower wrote: M on FICS and their adjudication system takes an extremely long time so I figured I might give them a better way. They must be doing one game at a time?
If you just want the end position analyzed, there is no need to use F13/F14, I can create a tool to extract end position of every game in a pgn then analyzed those positions using an engine, then those positions that are winning or losing by a given score threshold can be separated for example.
As Dann mentioned I also use pgn2fen to get the fen or epd, then scan it to get the last position of every game.
I use a simple program called leaf to do the same thing:

Code: Select all

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

char string[32767];

int main(void)
{
    while (fgets(string, sizeof string, stdin))
    {

        if (strstr(string," pm ")) continue;
        puts(string);
    }
    return 0;
}
User avatar
reflectionofpower
Posts: 1655
Joined: Fri Mar 01, 2013 5:28 pm
Location: USA

Re: Tool for converting PGN to EPD for analysis.

Post by reflectionofpower »

Ferdy wrote:
reflectionofpower wrote: M on FICS and their adjudication system takes an extremely long time so I figured I might give them a better way. They must be doing one game at a time?
If you just want the end position analyzed, there is no need to use F13/F14, I can create a tool to extract end position of every game in a pgn then analyzed those positions using an engine, then those positions that are winning or losing by a given score threshold can be separated for example.
As Dann mentioned I also use pgn2fen to get the fen or epd, then scan it to get the last position of every game.
if u could do that it would be fantastic!
"Without change, something sleeps inside us, and seldom awakens. The sleeper must awaken." (Dune - 1984)

Lonnie
Ferdy
Posts: 4851
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Tool for converting PGN to EPD for analysis.

Post by Ferdy »

Dann Corbit wrote:
Ferdy wrote:
reflectionofpower wrote: M on FICS and their adjudication system takes an extremely long time so I figured I might give them a better way. They must be doing one game at a time?
If you just want the end position analyzed, there is no need to use F13/F14, I can create a tool to extract end position of every game in a pgn then analyzed those positions using an engine, then those positions that are winning or losing by a given score threshold can be separated for example.
As Dann mentioned I also use pgn2fen to get the fen or epd, then scan it to get the last position of every game.
I use a simple program called leaf to do the same thing:

Code: Select all

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

char string[32767];

int main(void)
{
    while (fgets(string, sizeof string, stdin))
    {

        if (strstr(string," pm ")) continue;
        puts(string);
    }
    return 0;
}
Am using python script and when necessary together with existing tools like pgn2fen, pgn-extract, and Norman's tools.
User avatar
reflectionofpower
Posts: 1655
Joined: Fri Mar 01, 2013 5:28 pm
Location: USA

Re: Tool for converting PGN to EPD for analysis.

Post by reflectionofpower »

Dann Corbit wrote:
I always get the message:
"Error loading PGN: Error: Line 5157: Unexpected End-of-file"
for any PGN file.
It can be safely ignored, as the last PGN move is always included in the output file.
I did not put the output >, I got a EPD file but not at the end of all the games like I wanted, looks like i got epd for every single move on every single game
"Without change, something sleeps inside us, and seldom awakens. The sleeper must awaken." (Dune - 1984)

Lonnie
Dann Corbit
Posts: 12792
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Tool for converting PGN to EPD for analysis.

Post by Dann Corbit »

reflectionofpower wrote:
Dann Corbit wrote:
I always get the message:
"Error loading PGN: Error: Line 5157: Unexpected End-of-file"
for any PGN file.
It can be safely ignored, as the last PGN move is always included in the output file.
I did not put the output >, I got a EPD file but not at the end of all the games like I wanted, looks like i got epd for every single move on every single game
Yes, you have to filter for terminal book lines only.
That is what my sample script did.
The terminal book endpoints do not have a predicted move.

It is also interesting to analyze the interior nodes, because they often contain gaffes.

IOW, you might exit at a safe endpoint, but halfway down the line the opponent can go a rook up by making the right play.
Yes, that really does happen.
Ferdy
Posts: 4851
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Tool for converting PGN to EPD for analysis.

Post by Ferdy »

I will try to make one, here is the sample that I am going to do, check if I this is correct.
Get the pgn, convert it to epd with pm move from existing game, then take only the last position of that game, sample output format from pgn2fen.

Code: Select all

8/6k1/8/2N3p1/2B5/2n2KPP/1r3P2/8 b - - fmvn 58; hmvc 19; pm Kf6;
That fmvn is the full move number. The hmvc is the fifty move counter, this is important to be given to the engine for analysis, so it could be aware of drawn position by fifty move rule. The pm is the actual move played on that position. After extraction of all these end position, analyze it with a uci engine, user can choose the time, hash, threads and others. Say a threshold of 500 cp or more as winning and -500 or less as losing, these positions can be saved to win_los.epd for example.
The epd output format could be.

Code: Select all

8/6k1/8/2N3p1/2B5/2n2KPP/1r3P2/8 b - - fmvn 58; hmvc 19; pm Kf6; bm g7f6; acs 10; ce +600; c0 "Stockfish 6, 128mb, 1 thread";
Ferdy
Posts: 4851
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Tool for converting PGN to EPD for analysis.

Post by Ferdy »

Here is a working version.

A. Required files:
1. ai_engine.exe
>> just rename sf6.exe to this filename
2. pgn2fen.exe
>> for pgn to epd converstion
3. orig.pgn
>> your source pgn file

B. Output files:
1. sideToMoveWins.epd
2. sideToMoveLoss.epd
3. end_pos.epd
Contents of these files are overwritten in every run.
Sample output.

Code: Select all

6k1/2Q2p1p/2Pb2p1/8/1pB5/1N5P/1P4PK/4q3 w - - fmvn 41; hmvc 3; pm Qxd6; bm c7d6; ce 1729; c0 "Stockfish 6 64 POPCNT";
8/3N1pkp/3P2p1/pp6/7P/P7/2r4r/3R1K2 b - - fmvn 41; hmvc 15; pm Rh1#; bm h2h1; dm 1; c0 "Stockfish 6 64 POPCNT"
;

Sample run.

Code: Select all

Game end position analyzer

System        : Windows
Release       : 7
Machine       : AMD64
Processor     : Intel64 Family 6 Model 42 Stepping 7, GenuineIntel


orig.pgn file is found!!
pgn2fen.exe file is found!!
ai_engine.exe file is found!!


Enter 1 to see the log in screen [0/1]? 0
Enter engine analysis time in milliseconds? 300
Enter engine Hash value in MB? 128
Enter engine number of threads? 1
Enter win margin in cp? 500


Converting pgn to epd ...
Saving end position of every game ...
Analyzing positions ...

Starting engine ai_engine.exe ...
id name Stockfish 6 64 POPCNT


Pos 1: 5rk1/1p1q1pb1/2n3pp/4p3/2P3n1/6P1/1BN1PPBP/Q2R2K1 b - - fmvn 22; hmvc 2; pm Qf5;
fen: 5rk1/1p1q1pb1/2n3pp/4p3/2P3n1/6P1/1BN1PPBP/Q2R2K1 b - - 2 22
engine bestmove: d7f5
engine time (ms): 303
engine score (cp): 14 @depth 16


Pos 2: 2rr2k1/1p3p1p/5p2/pPpn3q/P4b2/2Q2N1P/4BPP1/1R3RK1 b - - fmvn 21; hmvc 1; pm Nxc3;
fen: 2rr2k1/1p3p1p/5p2/pPpn3q/P4b2/2Q2N1P/4BPP1/1R3RK1 b - - 1 21
engine bestmove: d5c3
engine time (ms): 300
engine score (cp): 1346 @depth 18


Pos 3: 8/1p6/p2k1n1p/3P1p2/PP1K1P1p/8/6B1/8 b - - fmvn 51; hmvc 17; pm Nh5;
fen: 8/1p6/p2k1n1p/3P1p2/PP1K1P1p/8/6B1/8 b - - 17 51
engine bestmove: b7b6
engine time (ms): 300
engine score (cp): 241 @depth 18


Pos 4: 3q1rn1/2p1p1k1/3p2pp/p2Pn3/Pr1NP3/8/3QB1PP/3R1RK1 w - - fmvn 27; hmvc 0; pm Ne6+;
fen: 3q1rn1/2p1p1k1/3p2pp/p2Pn3/Pr1NP3/8/3QB1PP/3R1RK1 w - - 0 27
engine bestmove: d4e6
engine time (ms): 300
engine score (cp): 1061 @depth 17


Pos 5: r3kbnr/ppq1pppp/2np4/2p5/6b1/5NPP/PPPPPPB1/RNBQ1RK1 w kq - fmvn 6; hmvc 1; pm hxg4;
fen: r3kbnr/ppq1pppp/2np4/2p5/6b1/5NPP/PPPPPPB1/RNBQ1RK1 w kq - 1 6
engine bestmove: h3g4
engine time (ms): 303
engine score (cp): 419 @depth 14


Pos 6: 8/7r/1k6/1p6/8/8/1R6/1K6 w - - fmvn 93; hmvc 12; pm Re2;
fen: 8/7r/1k6/1p6/8/8/1R6/1K6 w - - 12 93
engine bestmove: b2b3
engine time (ms): 300
engine score (cp): -6 @depth 29


Pos 7: 1n2r3/b5k1/1ppR2pp/p3P2n/P1B5/2P4P/5PP1/2B1R1K1 w - - fmvn 34; hmvc 1; pm g4;
fen: 1n2r3/b5k1/1ppR2pp/p3P2n/P1B5/2P4P/5PP1/2B1R1K1 w - - 1 34
engine bestmove: g2g4
engine time (ms): 304
engine score (cp): 916 @depth 14


Pos 8: r5k1/5p1p/p5r1/3q4/2np3Q/2P3P1/P4P2/3R2K1 b - - fmvn 31; hmvc 1; pm d3;
fen: r5k1/5p1p/p5r1/3q4/2np3Q/2P3P1/P4P2/3R2K1 b - - 1 31
engine bestmove: c4e5
engine time (ms): 301
engine score (cp): 1404 @depth 16


Pos 9: 5r1k/6p1/1P1R4/2p1qPn1/2P1p1p1/4P3/3QB1K1/8 w - - fmvn 40; hmvc 0; pm Bxg4;
fen: 5r1k/6p1/1P1R4/2p1qPn1/2P1p1p1/4P3/3QB1K1/8 w - - 0 40
engine bestmove: d6d8
engine time (ms): 307
engine score (cp): 333 @depth 16


Pos 10: 1R6/5pkp/4pPp1/2b1P3/6P1/6KP/r5B1/4r3 b - - fmvn 35; hmvc 0; pm Kh6;
fen: 1R6/5pkp/4pPp1/2b1P3/6P1/6KP/r5B1/4r3 b - - 0 35
engine bestmove: g7h6
engine time (ms): 309
engine score (cp): 1176 @depth 14


Pos 11: 1n3rk1/2q1p1bp/n2p2p1/p1pP1p2/PpN1PN2/2PQBP2/1P4PP/R4RK1 w - - fmvn 21; hmvc 6; pm Ne6;
fen: 1n3rk1/2q1p1bp/n2p2p1/p1pP1p2/PpN1PN2/2PQBP2/1P4PP/R4RK1 w - - 6 21
engine bestmove: f4e6
engine time (ms): 306
engine score (cp): 844 @depth 14


Pos 12: 4R3/p6p/3r1kp1/5p2/2q2P2/1p4PP/4R1K1/8 b - - fmvn 40; hmvc 3; pm Re6;
fen: 4R3/p6p/3r1kp1/5p2/2q2P2/1p4PP/4R1K1/8 b - - 3 40
engine bestmove: c4c6
engine time (ms): 301
engine score (cp): 1575 @depth 14


Pos 13: 6k1/2Q2p1p/2Pb2p1/8/1pB5/1N5P/1P4PK/4q3 w - - fmvn 41; hmvc 3; pm Qxd6;
fen: 6k1/2Q2p1p/2Pb2p1/8/1pB5/1N5P/1P4PK/4q3 w - - 3 41
engine bestmove: c7d6
engine time (ms): 300
engine score (cp): 1729 @depth 15


Pos 14: 8/3N1pkp/3P2p1/pp6/7P/P7/2r4r/3R1K2 b - - fmvn 41; hmvc 15; pm Rh1#;
fen: 8/3N1pkp/3P2p1/pp6/7P/P7/2r4r/3R1K2 b - - 15 41
engine bestmove: h2h1
engine time (ms): 1
engine score (dm): 1 @depth 127
Download.
http://www.mediafire.com/download/r6s4r ... tion-1.rar
User avatar
reflectionofpower
Posts: 1655
Joined: Fri Mar 01, 2013 5:28 pm
Location: USA

Re: Tool for converting PGN to EPD for analysis.

Post by reflectionofpower »

Ferdy wrote:Here is a working version.

A. Required files:
1. ai_engine.exe
>> just rename sf6.exe to this filename
2. pgn2fen.exe
>> for pgn to epd converstion
3. orig.pgn
>> your source pgn file

B. Output files:
1. sideToMoveWins.epd
2. sideToMoveLoss.epd
3. end_pos.epd
Contents of these files are overwritten in every run.
Sample output.

Code: Select all

6k1/2Q2p1p/2Pb2p1/8/1pB5/1N5P/1P4PK/4q3 w - - fmvn 41; hmvc 3; pm Qxd6; bm c7d6; ce 1729; c0 "Stockfish 6 64 POPCNT";
8/3N1pkp/3P2p1/pp6/7P/P7/2r4r/3R1K2 b - - fmvn 41; hmvc 15; pm Rh1#; bm h2h1; dm 1; c0 "Stockfish 6 64 POPCNT"
;


Download.
http://www.mediafire.com/download/r6s4r ... tion-1.rar
thanks, I just downloaded it, I just got home L fiddle with it later on or tomorrow.
"Without change, something sleeps inside us, and seldom awakens. The sleeper must awaken." (Dune - 1984)

Lonnie
User avatar
reflectionofpower
Posts: 1655
Joined: Fri Mar 01, 2013 5:28 pm
Location: USA

Re: Tool for converting PGN to EPD for analysis.

Post by reflectionofpower »

[quote="Ferdy"]Here is a working version.

A. Required files:
1. ai_engine.exe
>> just rename sf6.exe to this filename
2. pgn2fen.exe
>> for pgn to epd converstion
3. orig.pgn
>> your source pgn file

B. Output files:
1. sideToMoveWins.epd
2. sideToMoveLoss.epd
3. end_pos.epd
Contents of these files are overwritten in every run.
Sample output.

Code: Select all

6k1/2Q2p1p/2Pb2p1/8/1pB5/1N5P/1P4PK/4q3 w - - fmvn 41; hmvc 3; pm Qxd6; bm c7d6; ce 1729; c0 "Stockfish 6 64 POPCNT";
8/3N1pkp/3P2p1/pp6/7P/P7/2r4r/3R1K2 b - - fmvn 41; hmvc 15; pm Rh1#; bm h2h1; dm 1; c0 "Stockfish 6 64 POPCNT"
;

Sample run.

Code: Select all

Game end position analyzer

System        : Windows
Release       : 7
Machine       : AMD64
Processor     : Intel64 Family 6 Model 42 Stepping 7, GenuineIntel


orig.pgn file is found!!
pgn2fen.exe file is found!!
ai_engine.exe file is found!!


Enter 1 to see the log in screen [0/1]? 0
Enter engine analysis time in milliseconds? 300
Enter engine Hash value in MB? 128
Enter engine number of threads? 1
Enter win margin in cp? 500


Converting pgn to epd ...
Saving end position of every game ...
Analyzing positions ...

Starting engine ai_engine.exe ...
id name Stockfish 6 64 POPCNT


Pos 1: 5rk1/1p1q1pb1/2n3pp/4p3/2P3n1/6P1/1BN1PPBP/Q2R2K1 b - - fmvn 22; hmvc 2; pm Qf5;
fen: 5rk1/1p1q1pb1/2n3pp/4p3/2P3n1/6P1/1BN1PPBP/Q2R2K1 b - - 2 22
engine bestmove: d7f5
engine time (ms): 303
engine score (cp): 14 @depth 16


Pos 2: 2rr2k1/1p3p1p/5p2/pPpn3q/P4b2/2Q2N1P/4BPP1/1R3RK1 b - - fmvn 21; hmvc 1; pm Nxc3;
fen: 2rr2k1/1p3p1p/5p2/pPpn3q/P4b2/2Q2N1P/4BPP1/1R3RK1 b - - 1 21
engine bestmove: d5c3
engine time (ms): 300
engine score (cp): 1346 @depth 18
l][/quote]

I am getting a different output than you, ex.

r7/1pR3q1/2nP4/2k5/5Q2/p7/P1P1K3/8 w - - fmvn 38; hmvc 4; pm Kd2;
8/8/6n1/2p3P1/8/1Pk1K3/3N4/8 b - - fmvn 54; hmvc 8; pm Kb4;
7k/pq4b1/2p3p1/3p1P1p/4r3/P1P2QP1/1P5P/2B2R1K w - - fmvn 35; hmvc 1; pm fxg6;
8/8/1kp2n1p/4pB2/4PbP1/1KPP4/6r1/3R3R w - - fmvn 45; hmvc 17; pm Rhg1;
5rk1/3n1p1p/p3p1P1/3p4/1P3P2/P5R1/2r5/6RK b - - fmvn 34; hmvc 0; pm fxg6;
r1b1k1nr/Np1p1ppp/1q2p3/8/4P3/2N1b3/PPP2PPP/R2QKB1R b KQkq - fmvn 8; hmvc 0; pm Bxf2+;
8/3r2k1/p1Q4p/1pp1P1p1/4n3/1P4PP/P5BK/8 w - - fmvn 41; hmvc 2; pm Bxe4;


  this is from 7 positions

I had cp at 100, I wanted to have like a +1.5 would be a win, would this be 150?
"Without change, something sleeps inside us, and seldom awakens. The sleeper must awaken." (Dune - 1984)

Lonnie
Ferdy
Posts: 4851
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Tool for converting PGN to EPD for analysis.

Post by Ferdy »

[quote="reflectionofpower"][quote="Ferdy"]Here is a working version.

A. Required files:
1. ai_engine.exe
>> just rename sf6.exe to this filename
2. pgn2fen.exe
>> for pgn to epd converstion
3. orig.pgn
>> your source pgn file

B. Output files:
1. sideToMoveWins.epd
2. sideToMoveLoss.epd
3. end_pos.epd
Contents of these files are overwritten in every run.
Sample output.

Code: Select all

6k1/2Q2p1p/2Pb2p1/8/1pB5/1N5P/1P4PK/4q3 w - - fmvn 41; hmvc 3; pm Qxd6; bm c7d6; ce 1729; c0 "Stockfish 6 64 POPCNT";
8/3N1pkp/3P2p1/pp6/7P/P7/2r4r/3R1K2 b - - fmvn 41; hmvc 15; pm Rh1#; bm h2h1; dm 1; c0 "Stockfish 6 64 POPCNT"
;

Sample run.

Code: Select all

Game end position analyzer

System        : Windows
Release       : 7
Machine       : AMD64
Processor     : Intel64 Family 6 Model 42 Stepping 7, GenuineIntel


orig.pgn file is found!!
pgn2fen.exe file is found!!
ai_engine.exe file is found!!


Enter 1 to see the log in screen [0/1]? 0
Enter engine analysis time in milliseconds? 300
Enter engine Hash value in MB? 128
Enter engine number of threads? 1
Enter win margin in cp? 500


Converting pgn to epd ...
Saving end position of every game ...
Analyzing positions ...

Starting engine ai_engine.exe ...
id name Stockfish 6 64 POPCNT


Pos 1: 5rk1/1p1q1pb1/2n3pp/4p3/2P3n1/6P1/1BN1PPBP/Q2R2K1 b - - fmvn 22; hmvc 2; pm Qf5;
fen: 5rk1/1p1q1pb1/2n3pp/4p3/2P3n1/6P1/1BN1PPBP/Q2R2K1 b - - 2 22
engine bestmove: d7f5
engine time (ms): 303
engine score (cp): 14 @depth 16


Pos 2: 2rr2k1/1p3p1p/5p2/pPpn3q/P4b2/2Q2N1P/4BPP1/1R3RK1 b - - fmvn 21; hmvc 1; pm Nxc3;
fen: 2rr2k1/1p3p1p/5p2/pPpn3q/P4b2/2Q2N1P/4BPP1/1R3RK1 b - - 1 21
engine bestmove: d5c3
engine time (ms): 300
engine score (cp): 1346 @depth 18
[/quote]

I am getting a different output than you, ex.

r7/1pR3q1/2nP4/2k5/5Q2/p7/P1P1K3/8 w - - fmvn 38; hmvc 4; pm Kd2;
8/8/6n1/2p3P1/8/1Pk1K3/3N4/8 b - - fmvn 54; hmvc 8; pm Kb4;
7k/pq4b1/2p3p1/3p1P1p/4r3/P1P2QP1/1P5P/2B2R1K w - - fmvn 35; hmvc 1; pm fxg6;
8/8/1kp2n1p/4pB2/4PbP1/1KPP4/6r1/3R3R w - - fmvn 45; hmvc 17; pm Rhg1;
5rk1/3n1p1p/p3p1P1/3p4/1P3P2/P5R1/2r5/6RK b - - fmvn 34; hmvc 0; pm fxg6;
r1b1k1nr/Np1p1ppp/1q2p3/8/4P3/2N1b3/PPP2PPP/R2QKB1R b KQkq - fmvn 8; hmvc 0; pm Bxf2+;
8/3r2k1/p1Q4p/1pp1P1p1/4n3/1P4PP/P5BK/8 w - - fmvn 41; hmvc 2; pm Bxe4;


  this is from 7 positions[/quote]
The output that I have shown is from the console. What about yours?

[quote="reflectionofpower"]
I had cp at 100, I wanted to have like a +1.5 would be a win, would this be 150?[/quote]
Yes. When you run the program, you will be asked of the following. Input 150 in win margin.

[quote]Enter 1 to see the log in screen? 0
Enter engine analysis time in milliseconds? 300
Enter engine Hash value in MB? 128
Enter engine number of threads? 1
Enter win margin in cp? 500[/quote]
Were you able to run the executable file that I have uploaded?