To get the EPD predicted variation is GradualTest the best option?

[Dann Corbit]

Is have tested only the ana-sf.bat and the tool works like a charm! I can see that this file refers to the bench.txt file:

Code: Select all

bench 8192 4 60000 epd.epd movetime
quit

I could understand comparing with the version you resent after knowing my CPU that "4" stands for the number of the threads to use and "60000" means 60 seconds per move. So it is easy to modify those parameters according to needs. What is "8192"? I guess it is optimum setting - half of the RAM (from 16 GB I said I have installed)... right?

Yes, 8 GB RAM for analysis

I tested for 1 sec per move, just to see it is working. So far all is perfect! I will try later a long task.

What will be in future when newer SF versions will be released? Do you regularly post your modified SF versions anywhere like GitHub or this forum? Is it hard to learn what to modify in the SF source-code?

If you are not a programmer, it would be quite hard to stay current.
I generally perform an update with each new iteration of SF and other tools of interest.
Unfortunately, my MSYS2 installation died today, and I have not resurrected it yet so you have the current latest version of mine.
I often post my SF update to the link I mentioned that contains source code. That link should be reusable and you can just collect it without asking.
The new binary will be in a folder named \stockfish-master\src

[Dann Corbit]

Here is the link to my SF build for today:

[Ferdy]

I have tried to use GradualTest 2.6 to analyse different EPD files (some with few hundreds FENs). I want to get the pv=predicted variation just to verify the solutions of tactical puzzles or get best plans in strategical type of positions according to the engine. The tool is old and could not find the author, Mr. Odd Gunnar Malin, anywhere to kindly ask for some help. I find the tool awesome, but it works flawless only with Crafty (the old stable 25.2 version). Anything else I tried I could get only partial results. For example Stockfish 11 runs like half of a file with 300 FENs when suddenly crash and stop analyzing further, despite the positions are legal and could not find any reason for failure. Other UCI new engines on same test crash on other positions. "Fire" engine runs all... but give me only the bm=best move and not the pv. When running UCI engines I use the instructions from the tool and make an "eng" file with full path for the engine location and a separate batch file to run it and I don't know where I do any mistake. Maybe I do not know to run the new engines with this tool, or the tool does not support the new engines.

If Gradualtest is not my best option is there any other tool that will do the job? I want to mention that I want to get the pv in short notation (SAN).
Ideally would be a tool that the output from the engine analysis would be a PGN with FENs instead of games and the chess moves (the solutions) would be numbered, but I guess this tool does not exist yet.

Any suggestions?

I am not familiar with GradualTest, what epd format do you want to analyze? Can you give an example?
Outputting engine pv is not that difficult, can you give an example output format you need?

On this:

Ideally would be a tool that the output from the engine analysis would be a PGN with FENs instead of games and the chess moves (the solutions) would be numbered,

Can you give an example?

[Alex64]

Offtopic... Sorry I could not login to this site for long time. I thought the server went down, until I found I need a proxy located in USA.

With Gradualtest you get what it is called "a decorated EPD" (of course might be hundreds of chess positions inside the file):

Code: Select all

r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq - acd 9; acn 13005735; acs 11; ce -55; fmvn 1; pm Nc3; pv Nc3 Nf6 Nd5 O-O Bg5 Bg4 Nxb6 axb6 d5;
r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq - acd 11; acn 18540833; acs 10; ce -8; fmvn 1; pm Nh6; pv Nh6 Nxf7 Bb4+ c3 dxc3 bxc3 Nxf7 Bxf7+ Kxf7 cxb4 Qf6 Qb3+ d5 Nc3 Rd8 exd5;

I can cannibalize the "decoration" in Notepad++ (for example) with find/replace for a string as "acd.?pv" (without quotation marks) which will remove all between acd and pv and get the result:

Code: Select all

r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq -  Nc3 Nf6 Nd5 O-O Bg5 Bg4 Nxb6 axb6 d5;
r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq -  Nh6 Nxf7 Bb4+ c3 dxc3 bxc3 Nxf7 Bxf7+ Kxf7 cxb4 Qf6 Qb3+ d5 Nc3 Rd8 exd5;

Using the software from Mr. Dann Corbit (BIG THANKS!) I can use a stronger engine than Crafty. The resulted "decoration" is similar, just having to remove an extra tag (bm= best move) located at the end. I can apply a similar REGEX string to get rid of it.

What I wonder, if possible, might need too much work of programming. I would like to get this:

Code: Select all

r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq -  1.Nc3 Nf6 2.Nd5 O-O 3.Bg5 Bg4 4.Nxb6 axb6 5.d5;
r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq -  1...Nh6 2.Nxf7 Bb4+ 3.c3 dxc3 4.bxc3 Nxf7 5.Bxf7+ Kxf7 6.cxb4 Qf6 7.Qb3+ d5 Nc3 8.Rd8 exd5;

Even better would be a fake PGN. Which is a PGN that actually contains FENs (like an EPD) instead of games, and it would look like this (to use the same examples):

Code: Select all

[Event "XXX"]
[Site "YYY"]
[Date "????.??.??"]
[Round "?"]
[White "001"]
[Black "?"]
[Result "*"]
[Source "ZZZ"]
[FEN "r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq -"]

1.Nc3 Nf6 2.Nd5 O-O 3.Bg5 Bg4 4.Nxb6 axb6 5.d5;

[Event "XXX"]
[Site "YYY"]
[Date "????.??.??"]
[Round "?"]
[White "002"]
[Black "?"]
[Result "*"]
[Source "ZZZ"]
[FEN "r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq -"]

1...Nh6 2.Nxf7 Bb4+ 3.c3 dxc3 4.bxc3 Nxf7 5.Bxf7+ Kxf7 6.cxb4 Qf6 7.Qb3+ d5 Nc3 8.Rd8 exd5;

Why would I want a fake PGN? Many GUI chess app do not open EPD with solutions to see them, but they do open PGNs and can be tricked to open the fake ones. For example Lucas chess app, or Kvetka... not sure about CB products which might be sensible to open only perfect PGNs.

[Dann Corbit]

I made a new thing called pv2pgn. You can get it here:


You use it like this:
prse top < SearchLog.txt > sl
pv2pgn < sl > slcrude.pgn
pgn-extract -s < slcrude.pgn > sl.pgn

You can get pgn-extract from here:
https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/

The file sl looks like this:

Code: Select all

r2q1rk1/1p2bppp/p2pbn2/P3p3/2B1P3/1NN1B3/1PPQ1PPP/R4RK1 b - - acd 36; ce 0; acs 7; acn 581514; pv Bxc4 Rfd1 Rc8 Nc1 d5 Bb6 Qe8 exd5 Bb4 d6 Bxc3 bxc3 Qd7 Qe1 Bd5 Nd3 Qxd6 Qxe5 Qxe5 Nxe5 Rxc3 Rac1 Be4 f3 Bxc2 Rd2 Rfc8 Bd4 R3c7 Bb6 ; bm Bxc4;
1r2r1k1/1p1b1pbp/p2p2p1/q1pPn2n/4P1P1/N1N4P/1P3PB1/R1BQ1RK1 b - - acd 36; ce 0; acs 12; acn 993456; pv Nf6 f4 Nfxg4 hxg4 Nxg4 f5 Ne5 Bg5 Qc7 f6 Bh8 Nc2 a5 Ne3 Qd8 Qe1 h6 Bxh6 Bxf6 Rxf6 Qxf6 Qg3 b5 Nf5 Bxf5 exf5 Qxf5 Ne4 Nd7 Rf1 Qe5 Bf4 Qxb2 Nxd6 Qd4+ Kh2 Qh8+ Kg1 ; bm Nf6;
r2qk2r/p1pbbppp/2p5/3pP3/3Nn3/5P2/PPP3PP/RNBQ1RK1 b kq - acd 36; ce 0; acs 4; acn 319779; pv Nc5 f4 O-O f5 Ne4 Nc3 Bc5 Na4 Be7 ; bm Nc5;

...

The file slcrude.pgn looks like this:

Code: Select all

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[Setup "1"]
[FEN "r2q1rk1/1p2bppp/p2pbn2/P3p3/2B1P3/1NN1B3/1PPQ1PPP/R4RK1 b - - 0 1"]

1. ... Bxc4 Rfd1 Rc8 Nc1 d5 Bb6 Qe8 exd5 Bb4 d6 Bxc3 bxc3 Qd7 Qe1 Bd5 Nd3 Qxd6 Qxe5 Qxe5 Nxe5 Rxc3 Rac1 Be4 f3 Bxc2 Rd2 Rfc8 Bd4 R3c7 Bb6  *


[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[Setup "1"]
[FEN "1r2r1k1/1p1b1pbp/p2p2p1/q1pPn2n/4P1P1/N1N4P/1P3PB1/R1BQ1RK1 b - - 0 1"]

1. ... Nf6 f4 Nfxg4 hxg4 Nxg4 f5 Ne5 Bg5 Qc7 f6 Bh8 Nc2 a5 Ne3 Qd8 Qe1 h6 Bxh6 Bxf6 Rxf6 Qxf6 Qg3 b5 Nf5 Bxf5 exf5 Qxf5 Ne4 Nd7 Rf1 Qe5 Bf4 Qxb2 Nxd6 Qd4+ Kh2 Qh8+ Kg1  *


[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[Setup "1"]
[FEN "r2qk2r/p1pbbppp/2p5/3pP3/3Nn3/5P2/PPP3PP/RNBQ1RK1 b kq - 0 1"]

1. ... Nc5 f4 O-O f5 Ne4 Nc3 Bc5 Na4 Be7  *


[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[Setup "1"]
[FEN "r1r3k1/pp1bppbp/3p1np1/q7/2BNP2P/2N1BP2/PPPQ2P1/1K1R3R b - - 0 1"]

1. ... Rxc4 Nb3 Qc7 g4 h5 gxh5 Rxc3 Qxc3 Qxc3 bxc3 Nxh5 Bd4 Bf8 Nc1 Bh6 Rhe1 b6 Bf2 Bb5 Kb2 Rc8 Nd3 Kf8 Bd4 Bd7 Kb3 Be6+ Ka3 Bd7 Kb2 Bb5 Rg1 Bd7 Rde1 Be8 Nb4 Ba4 Rg2 e6 e5 dxe5 Bxe5 Bb5  *

...

And the file sl.pgn looks like this:

Code: Select all

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[SetUp "1"]
[FEN "r2q1rk1/1p2bppp/p2pbn2/P3p3/2B1P3/1NN1B3/1PPQ1PPP/R4RK1 b - - 0 1"]
[Setup "1"]

1... Bxc4 2. Rfd1 Rc8 3. Nc1 d5 4. Bb6 Qe8 5. exd5 Bb4 6. d6 Bxc3 7. bxc3
Qd7 8. Qe1 Bd5 9. Nd3 Qxd6 10. Qxe5 Qxe5 11. Nxe5 Rxc3 12. Rac1 Be4 13. f3
Bxc2 14. Rd2 Rfc8 15. Bd4 R3c7 16. Bb6 *

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "*"]
[SetUp "1"]
[FEN "1r2r1k1/1p1b1pbp/p2p2p1/q1pPn2n/4P1P1/N1N4P/1P3PB1/R1BQ1RK1 b - - 0 1"]
[Setup "1"]

1... Nf6 2. f4 Nfxg4 3. hxg4 Nxg4 4. f5 Ne5 5. Bg5 Qc7 6. f6 Bh8 7. Nc2 a5
8. Ne3 Qd8 9. Qe1 h6 10. Bxh6 Bxf6 11. Rxf6 Qxf6 12. Qg3 b5 13. Nf5 Bxf5
14. exf5 Qxf5 15. Ne4 Nd7 16. Rf1 Qe5 17. Bf4 Qxb2 18. Nxd6 Qd4+ 19. Kh2
Qh8+ 20. Kg1 *

...

I will leave it to you to figure out how to remove the extraneous [Setup "1"] tag. The pgn standard says it has to be there, as silly as that may sound, but for some reason pgn-extact adds another

[Ferdy]

Offtopic... Sorry I could not login to this site for long time. I thought the server went down, until I found I need a proxy located in USA.

With Gradualtest you get what it is called "a decorated EPD" (of course might be hundreds of chess positions inside the file):

Code: Select all

r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq - acd 9; acn 13005735; acs 11; ce -55; fmvn 1; pm Nc3; pv Nc3 Nf6 Nd5 O-O Bg5 Bg4 Nxb6 axb6 d5;
r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq - acd 11; acn 18540833; acs 10; ce -8; fmvn 1; pm Nh6; pv Nh6 Nxf7 Bb4+ c3 dxc3 bxc3 Nxf7 Bxf7+ Kxf7 cxb4 Qf6 Qb3+ d5 Nc3 Rd8 exd5;

I can cannibalize the "decoration" in Notepad++ (for example) with find/replace for a string as "acd.?pv" (without quotation marks) which will remove all between acd and pv and get the result:

Code: Select all

r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq -  Nc3 Nf6 Nd5 O-O Bg5 Bg4 Nxb6 axb6 d5;
r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq -  Nh6 Nxf7 Bb4+ c3 dxc3 bxc3 Nxf7 Bxf7+ Kxf7 cxb4 Qf6 Qb3+ d5 Nc3 Rd8 exd5;

Using the software from Mr. Dann Corbit (BIG THANKS!) I can use a stronger engine than Crafty. The resulted "decoration" is similar, just having to remove an extra tag (bm= best move) located at the end. I can apply a similar REGEX string to get rid of it.

What I wonder, if possible, might need too much work of programming. I would like to get this:

Code: Select all

r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq -  1.Nc3 Nf6 2.Nd5 O-O 3.Bg5 Bg4 4.Nxb6 axb6 5.d5;
r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq -  1...Nh6 2.Nxf7 Bb4+ 3.c3 dxc3 4.bxc3 Nxf7 5.Bxf7+ Kxf7 6.cxb4 Qf6 7.Qb3+ d5 Nc3 8.Rd8 exd5;

Even better would be a fake PGN. Which is a PGN that actually contains FENs (like an EPD) instead of games, and it would look like this (to use the same examples):

Code: Select all

[Event "XXX"]
[Site "YYY"]
[Date "????.??.??"]
[Round "?"]
[White "001"]
[Black "?"]
[Result "*"]
[Source "ZZZ"]
[FEN "r1bqk1nr/ppp2ppp/1bnp4/8/2BPP3/5N2/P4PPP/RNBQ1RK1 w kq -"]

1.Nc3 Nf6 2.Nd5 O-O 3.Bg5 Bg4 4.Nxb6 axb6 5.d5;

[Event "XXX"]
[Site "YYY"]
[Date "????.??.??"]
[Round "?"]
[White "002"]
[Black "?"]
[Result "*"]
[Source "ZZZ"]
[FEN "r1bqk1nr/pppp1ppp/2n5/2b3N1/2BpP3/8/PPP2PPP/RNBQK2R b KQkq -"]

1...Nh6 2.Nxf7 Bb4+ 3.c3 dxc3 4.bxc3 Nxf7 5.Bxf7+ Kxf7 6.cxb4 Qf6 7.Qb3+ d5 Nc3 8.Rd8 exd5;

Why would I want a fake PGN? Many GUI chess app do not open EPD with solutions to see them, but they do open PGNs and can be tricked to open the fake ones. For example Lucas chess app, or Kvetka... not sure about CB products which might be sensible to open only perfect PGNs.

Created a command line tool using python-chess module. This is how it would work so far.
You feed the EPD file in the tool, you also input the engine and analysis time per pos.

There are 2 output file, pgn and epd.

Code: Select all

PS D:\pyenv\epdanalysistopgn> ls


    Directory: D:\pyenv\epdanalysistopgn


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       29/05/2020   7:22 am                .idea
d-----       29/05/2020   5:48 am                engine
d-----       29/05/2020   6:25 am                epd
d-----       29/05/2020   5:32 am                venv
-a----       29/05/2020   9:06 am           4098 eap.py


PS D:\pyenv\epdanalysistopgn> python.exe eap.py --help
usage: eap.py [-h] --input INPUT [--outputpgn OUTPUTPGN] [--outputepd OUTPUTEPD] --engine ENGINE --movetimems MOVETIMEMS

optional arguments:
  -h, --help            show this help message and exit
  --input INPUT         input epd file
  --outputpgn OUTPUTPGN
                        output pgn file in append mode, default=out.pgn
  --outputepd OUTPUTEPD
                        output epd file in append mode, default=out.epd
  --engine ENGINE       input engine file
  --movetimems MOVETIMEMS
                        input analysis time in ms, default=1000
                        
PS D:\pyenv\epdanalysistopgn> python.exe eap.py --input ./epd/wacnew.epd --outputpgn out_wacnew.pgn --outputepd out_wacnew.epd --engine ./engine/stockfish_11.exe --movetimems 1000                                                                                                                                       
Analysis starts ...
Analysis done!
Total Pos analyzed: 300
Timems/pos: 1000
Elapse: 0h:05m:00s:616ms
Mean timems/pos: 1002
TimeOverheadMs/pos: 2

Example pgn output:

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "?"]
[Black "?"]
[Result "1-0"]
[AnalysisMovetimeMs "1000"]
[Annotator "Stockfish 11 64 POPCNT"]
[EPDId "WAC.001"]
[FEN "2rr3k/pp3pp1/1nnqbN1p/3pN3/2pP4/2P3Q1/PPB4P/R4RK1 w - - 0 1"]
[SetUp "1"]

1. Qg6 Nxe5 2. Qh7# 1-0

Example epd output:

Code: Select all

2rr3k/pp3ppQ/1n1qbN1p/3pn3/2pP4/2P5/PPB4P/R4RK1 b - - acd 245; acs 1; ce 31998; id WAC.001; pm Qg6; pv Qg6 Nxe5 Qh7#; c0 "analyzed by Stockfish 11 64 POPCNT";

Is the output enough for you?

If you have python 3.8, and python-chess module installed that is better, if not I will just include the windows exe in the pack.

You can request other features, will implement it if possible.

I will be adding engine option in the command line, will release a beta later in github. If beta is good, I will add parallel option, your epd must be all legal and the tool should use a stable uci engine like Stockfish.

I name this tool "EAP - EPD Analysis to PGN". You may suggest a name for it.

[Alex64]

@Dann: I cannot download from your link on google drive.. it is locked... could you, please, post pv2pgn in other host site?

@Ferdy: I don't have the "python-chess module"...so, yes, please send me a link. And I am looking forward to see the beta release in github if you intend to develop it... would be nice.

[Dann Corbit]

Try again, I don't know why the file was not released the first time, I must have clicked something wrong. That never happened before, so far as I can recall

[Ferdy]

@Ferdy: I don't have the "python-chess module"...so, yes, please send me a link. And I am looking forward to see the beta release in github if you intend to develop it... would be nice.

Try the windows exe at https://github.com/fsmosca/EPD-Analyzer ... /v0.2.beta

Command line sample in Github repo is at https://github.com/fsmosca/EPD-Analyzer

[Ferdy]

Released v0.4.beta that fixes EPD output.
https://github.com/fsmosca/EPD-Analyzer/releases