finding bad evaluations by scaning pgn games

Discussion of chess software programming and technical issues.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
cdani
Posts: 2198
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

finding bad evaluations by scaning pgn games

Post by cdani » Thu Nov 10, 2016 6:33 pm

Maybe someone knows about any tool that is able to scan cutechess like pgn files so to expose the biggest differences of evaluations from one move to the next.

The idea is for example play some games between Stockfish and Andscacs, and then to be able to find positions where for example Andscacs evaluates as 0.20 and Stockfish in his tourn finds that is 1.3. Then one can review his evaluation or search to try to improve them. Many thanks.

Ferdy
Posts: 4583
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: finding bad evaluations by scaning pgn games

Post by Ferdy » Fri Nov 11, 2016 3:22 am

cdani wrote:Maybe someone knows about any tool that is able to scan cutechess like pgn files so to expose the biggest differences of evaluations from one move to the next.

The idea is for example play some games between Stockfish and Andscacs, and then to be able to find positions where for example Andscacs evaluates as 0.20 and Stockfish in his tourn finds that is 1.3. Then one can review his evaluation or search to try to improve them. Many thanks.
Nice idea :), just created a tool as this is also good for my engines.

It reads the cute chess pgn file and output interesting positions in the epd file.
If my opponent won the game and in those positions my engine's score is different from opp score by a certain margin say 0.5 pawn, save the epd.

Example 1:

Code: Select all

4r1k1/1p1q2p1/p1p5/P2p1r2/1Q1P4/1R2PR1P/1P4K1/8 w - - c0 "D368 {+0.00} - Sf7 {-1.82}"; c1 "Result 0-1"; c2 "file sf_games.pgn, game 3, move 50";
The score after the engine name in the c0 opcode is WPOV, meaning if positive, better for white if negative, better for black.
Here Sf7 (as black) which won the game (see c1 opcode), has a score of -1.82 better for black, but my engine only says 0.00, so this position is interesting to study for me. The game and move number can be tracked in c2 opcode.

Example 2:

Code: Select all

r5k1/1r6/1p1p1p2/1ppPpP1p/1P2P1p1/P2P2Pq/6RP/1R1Q2K1 b - - c0 "Sf7 {+0.25} - D368 {-0.85}"; c1 "Result 1-0"; c2 "file sf_games.pgn, game 16, move 34";
My engine likes its position but Sf likes its position too, but we already know where this would end up :).

Example 3:

Code: Select all

br1nr1k1/2pqbpp1/p2pp3/1p6/1P1PPP1P/PNPQR3/3N2BP/2R3K1 b - - c0 "Sf7 {+1.37} - D368 {+0.78}"; c1 "Result 1-0"; c2 "file sf_games.pgn, game 24, move 22";
Sf7 says I am winning, my engine says no you are just better.

You are just asks 3 questions:

Code: Select all

enter pgn file? sf_games.pgn
enter engine name? D368
enter score margin in pawn unit? 0.5
It will only consider 1-0 and 0-1 results. Positions are only saved when my engine's score is within +/-6.0 pawns.
Check if the info in the epd output are enough.

If this looks fine with you I will upload the pack.

User avatar
cdani
Posts: 2198
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: finding bad evaluations by scaning pgn games

Post by cdani » Fri Nov 11, 2016 5:50 am

Sure!!! Is more that good tool. Many thanks :-) Sure more people will use it!

Dann Corbit
Posts: 12125
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: finding bad evaluations by scaning pgn games

Post by Dann Corbit » Fri Nov 11, 2016 6:07 am

This is a very nice tool. The kanguruh author who wrote the online interface had a tool for that ut it as very fussy and sometimes did not work. But it did draw a nice picture of the eval discrepancy. I guess that many people can benefit from it and it might be used to search for test problems in an automated way
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
Guenther
Posts: 3938
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: finding bad evaluations by scaning pgn games

Post by Guenther » Fri Nov 11, 2016 11:09 am

cdani wrote:Maybe someone knows about any tool that is able to scan cutechess like pgn files so to expose the biggest differences of evaluations from one move to the next.

The idea is for example play some games between Stockfish and Andscacs, and then to be able to find positions where for example Andscacs evaluates as 0.20 and Stockfish in his tourn finds that is 1.3. Then one can review his evaluation or search to try to improve them. Many thanks.
IMO by far the best tool for this is Toms Game Analyser.
http://home.pacific.net.au/~tommyinoz/gameanalyser.html
The one Dann mentioned in his post, if he had errors then probably because
the VB dlls were not correctly installed or registered in his WIN version.
I had no errors here in Win 7 64 after reinstalling the updated version
from 2009/09. (I can even use it portable if I copy the installation elsewhere)

I also want to add that the description is a bit outdated therefore people
may be didn't use it as much as it deserved.
There is NO NEED to convert DEBUG files (that is a historical feature!).
It works immediately from pgn files.

Note that cutechess pgns must be preprocessed though, but it is very
easy and a good editor with regex is sufficient.
This could be done with a script/batch too. It only needs 3 tiny steps.
The instructions below are adapted for my ultraedit and must be
changed for the used editor.

Code: Select all

//adapt cutechess PGN for Game Analyser:
//ultraedit style (^p = new line, ^(^) = set regex as variable
----------------------------------------------------------------------
1. convert UNIX => DOS
2. '^([0-9]^)s}' => '^1}'
3. 's,' => '' OR 's,' => }^p{

Code: Select all

//explanation
1. cute saves pgn to unix style txt format (= different line breaks/new lines), TGA needs DOS formatting for working
2. cute adds 's' for seconds in the move comment
3.a cute adds ',' and includes game end comment into the last move comment
3.b I don't like this mixing and so I divide both comments
and write the game end comment to a new line, which looks cleaner
After this the pgn is completely compatible with the Game Analyser
and it is a very mighty tool!
E.g. you can even give a range (interval) of moves to compare and not only
compare White/Black but also White/White and Black/Black instead...


The found positions can even be exported to text and epdsuite files.
Below is a snippet for the game from the screenhots.

Code: Select all

r2r2k1/pp1Pb1pp/4p3/2P2pN1/8/3QB3/q3BPPP/1R4K1 b - - sm Bxg5; c0 "Stockfish_14031419 vs Senpai_10 Event=CuteTest Date=2014.03.28 Site=RWBC Eval=-0,71";
r2r2k1/pR1P2pp/4p3/2P2pb1/8/3QB3/q3BPPP/6K1 b - - sm Bxe3; c0 "Stockfish_14031419 vs Senpai_10 Event=CuteTest Date=2014.03.28 Site=RWBC Eval=-1,03";
r2r2k1/pR1P2pp/4p3/2P2p2/8/4Q3/q3BPPP/6K1 b - - sm Qd5; c0 "Stockfish_14031419 vs Senpai_10 Event=CuteTest Date=2014.03.28 Site=RWBC Eval=-0,97";
r2r2k1/pR1P2pp/4p3/2Pq1p2/8/4QB2/5PPP/6K1 b - - sm Qa2; c0 "Stockfish_14031419 vs Senpai_10 Event=CuteTest Date=2014.03.28 Site=RWBC Eval=-1,52";
r2r2k1/pR1P2pp/4p3/2P2p2/8/4QBP1/q4P1P/6K1 b - - sm Qc4; c0 "Stockfish_14031419 vs Senpai_10 Event=CuteTest Date=2014.03.28 Site=RWBC Eval=-2,27";
Examples: (probably self explaining)
screenshot 1
Image
screenshot 2
Image
screenshot 3
Image
screenshot 4
Image

If you have any questions I am willing to answer.
(AFAIK Ferdinand also uses this tool from time to time...)

May be I should post this also to the regular forum, as it could be of general interest?

Guenther

Ferdy
Posts: 4583
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: finding bad evaluations by scaning pgn games

Post by Ferdy » Sun Nov 13, 2016 1:17 pm

cdani wrote:Sure!!! Is more that good tool. Many thanks :-) Sure more people will use it!
Place the pgn file and the exe program in same dir.
The output epd file is overwrite mode.

https://drive.google.com/file/d/0BwAOsu ... sp=sharing

Ferdy
Posts: 4583
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: finding bad evaluations by scaning pgn games

Post by Ferdy » Sun Nov 13, 2016 1:22 pm

Guenther wrote:
cdani wrote:Maybe someone knows about any tool that is able to scan cutechess like pgn files so to expose the biggest differences of evaluations from one move to the next.

The idea is for example play some games between Stockfish and Andscacs, and then to be able to find positions where for example Andscacs evaluates as 0.20 and Stockfish in his tourn finds that is 1.3. Then one can review his evaluation or search to try to improve them. Many thanks.
IMO by far the best tool for this is Toms Game Analyser.
http://home.pacific.net.au/~tommyinoz/gameanalyser.html
This is indeed a good tool but need pre-processing of the input pgn file.

User avatar
cdani
Posts: 2198
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: finding bad evaluations by scaning pgn games

Post by cdani » Sun Nov 13, 2016 8:32 pm

Thanks Guenther! I will review this also :-)

Post Reply