Put that PGN into the MRI

[Rebel]

Some snippets

MRI is a PGN utility for engine developers and/or beta testers to extract critical information from an eng-eng match, even from rating list PGN downloads with multiple engines such as CCRL, CEGT and probably others.

...

7. Lost Games Analysis - This function tries (emphasis added) to pinpoint the move where it all went wrong. It checks the 4 last moves and if the scores from move to move slowly but steady go down and down until the margin (default 1.50) is reached then it makes sense to have a look at that game fragment. While it's far from perfect this function probably is the most useful of the MRI utility.

...

This option 5c is one of the better features of MRI, download the annotated games of CCRL and or CEGT of your latest version and get the results of all games played against all opponents.

...

http://rebel13.nl/download/mri.html

[Guenther]

I downloaded it and tested it on a fresh pgn and have a few suggestions and questions.
This was a test of SF11 vs. SF11x2 (approx. half time used)

1. Unluckily it cuts off the name in the general overview, thus both entities look now the same (could be cured by allowing more chars I guess)
From the results and avg. depths I can still recognize who is who though.

2. It gives wrong values for book depth (we had this already for another tool of yours, if you remember?), the book is exactly 6 plies always and created by me. The time output also says 0:00 for one program. BTW should this be accumulated time or average time, I see no explanation in the output?

3. How does it parse and calculate the suspect openings? There is no record given, despite I know there are several entries with 1.00 margin (mostly for being extremely fast games and search gives too pessimistic evals at lower depths). Example below.

The pgn was created by cutechess-cli (it was correctly assumed), but modified by a script by me, which removes 's' at the end of the move times
and creates perfect columns. Also it is a slightly modified cutechess compiled by me, which adds at least 2 digits precision for the saved move times. May be this confuses MRI?

Here an example - they are all that way. (left the pgn headers out here)

Code: Select all

1. c4 {book} e5 {book}
2. Nc3 {book} f5 {book}
3. g3 {book} Nf6 {book}
4. e3 {+1.05/16 0.92} Nc6 {-0.30/17 1.92}
5. d4 {+0.93/14 0.25} Bb4 {+0.01/13 0.15}

BTW the one probably missed win is a nice find of a festung, if I am not mistaken.
[d]8/2p5/8/3k4/p7/B1R5/K7/3q4 b

Part of the MRI output referred above:

Code: Select all

Assume CuteChess PGN.

Current Settings:
Margin for suspect Openings          : 1.00
Margin for crazy games               : 1.00
Margin for possible missed won games : 5.00
Drop in Score Margin                 : 1.50
Phase Margin                         : 0.50
Lost Game Score Margin               : 1.50
Opening Repertory                    : on 
Double Game Detection                : off

Phase          Won Games (numbers)           Late Endgame             Match
Overview       MIDG END1 END2 END3      QUEEN ROOK LIGHT PAWN         Score
Stockfish_11    150    7   96   23          0   17    14   14   713.5 / 1000 (71.3%)
Stockfish_11     22    1   14    3          0    3     0    4   286.5 / 1000 (28.6%)

Phase              Won games %               Late Endgame             Match
Overview       MIDG END1 END2 END3      QUEEN ROOK LIGHT PAWN         Score
Stockfish_11   87.2 87.5 87.3 88.5        0.0 85.0  100% 77.8   713.5 / 1000 (71.3%)
Stockfish_11   12.8 12.5 12.7 11.5        0.0 15.0   0.0 22.2   286.5 / 1000 (28.6%)

Depths         MIDG END1 END2 END3     BOOK             TIME
Stockfish_11   17.1 17.5 22.9 25.2     3.5 (moves)     20:31
Stockfish_11   15.3 15.6 20.0 21.7     4.1 (moves)      0:00

Suspect opening lines overview (1.00)  View

Crazy games (incompatible scores)  (1.00)  View

Possible missed wins overview (5.00)  View
Stockfish_11-64 possibly missed a win in game 928 with a score of 5.26 (move 71... Qd2+) draw only.  Stockfish
8/2p5/8/3k4/p7/B1R5/K7/3q4 b - - sm Qd2+; acd 20; ce 5.26; acs 0.0s; 


Score drop overview (1.50)  View
Drop in score of 1.74 (0.00 to -1.74) for Stockfish_11x2-64 in game 3 during moves 18-19, game lost  Stockfish
rr4k1/1q1nbpp1/p1pp1nbp/4P3/3P1B2/P1N5/BP1Q1NPP/3R1RK1 b - - sm Nh5; acd 13; ce -1.74; acs 0.0s; 

[Rebel]

I downloaded it and tested it on a fresh pgn and have a few suggestions and questions.
This was a test of SF11 vs. SF11x2 (approx. half time used)

1. Unluckily it cuts off the name in the general overview, thus both entities look now the same (could be cured by allowing more chars I guess)
From the results and avg. depths I can still recognize who is who though.

The current length of the names is fixed and set to 12, I will make it user defined.

2. It gives wrong values for book depth (we had this already for another tool of yours, if you remember?), the book is exactly 6 plies always and created by me. The time output also says 0:00 for one program. BTW should this be accumulated time or average time, I see no explanation in the output?

I have an idea about the why, will give it attention.

3. How does it parse and calculate the suspect openings? There is no record given, despite I know there are several entries with 1.00 margin (mostly for being extremely fast games and search gives too pessimistic evals at lower depths). Example below.

An engine may play the first move with a score of -5.xx, it's only recorded if it loses the game.

The pgn was created by cutechess-cli (it was correctly assumed), but modified by a script by me, which removes 's' at the end of the move times
and creates perfect columns. Also it is a slightly modified cutechess compiled by me, which adds at least 2 digits precision for the saved move times. May be this confuses MRI?

Yes.

The "s" is typical for the Cute format, else it is assumed more Arena alike, thus full seconds only. Run the original PGN instead.

Here an example - they are all that way. (left the pgn headers out here)

Code: Select all

1. c4 {book} e5 {book}
2. Nc3 {book} f5 {book}
3. g3 {book} Nf6 {book}
4. e3 {+1.05/16 0.92} Nc6 {-0.30/17 1.92}
5. d4 {+0.93/14 0.25} Bb4 {+0.01/13 0.15}

BTW the one probably missed win is a nice find of a festung, if I am not mistaken.

Correct.

[Guenther]

I downloaded it and tested it on a fresh pgn and have a few suggestions and questions.
This was a test of SF11 vs. SF11x2 (approx. half time used)

1. Unluckily it cuts off the name in the general overview, thus both entities look now the same (could be cured by allowing more chars I guess)
From the results and avg. depths I can still recognize who is who though.

The current length of the names is fixed and set to 12, I will make it user defined.

2. It gives wrong values for book depth (we had this already for another tool of yours, if you remember?), the book is exactly 6 plies always and created by me. The time output also says 0:00 for one program. BTW should this be accumulated time or average time, I see no explanation in the output?

I have an idea about the why, will give it attention.

3. How does it parse and calculate the suspect openings? There is no record given, despite I know there are several entries with 1.00 margin (mostly for being extremely fast games and search gives too pessimistic evals at lower depths). Example below.

An engine may play the first move with a score of -5.xx, it's only recorded if it loses the game.

The pgn was created by cutechess-cli (it was correctly assumed), but modified by a script by me, which removes 's' at the end of the move times
and creates perfect columns. Also it is a slightly modified cutechess compiled by me, which adds at least 2 digits precision for the saved move times. May be this confuses MRI?

Yes.

The "s" is typical for the Cute format, else it is assumed more Arena alike, thus full seconds only. Run the original PGN instead.

Here an example - they are all that way. (left the pgn headers out here)

Code: Select all

1. c4 {book} e5 {book}
2. Nc3 {book} f5 {book}
3. g3 {book} Nf6 {book}
4. e3 {+1.05/16 0.92} Nc6 {-0.30/17 1.92}
5. d4 {+0.93/14 0.25} Bb4 {+0.01/13 0.15}

BTW the one probably missed win is a nice find of a festung, if I am not mistaken.

Correct.

Thanks for the answers!

I am still not sure about point 3., shouldn't it report then a 'suspect openings', if e.g. eval > 1.0 for White and White wins?
Or does it consider the eval of the reply too?
This would be a big improvement anyway. What, if you just record it as 'suspect', if the eval is >X for a few moves (let's say 3 or 4 again)?

[Guenther]

After the change in the pgn I got at least two different time results compared to before.
But still something there seems to be wrong? The test lasted over 16:03:xx.

Code: Select all

after adding (s) again to move times:

Depths         MIDG END1 END2 END3     BOOK             TIME
Stockfish_11   17.1 17.5 22.9 25.2     3.5 (moves)   5:09:40
Stockfish_11   15.3 15.6 20.0 21.7     4.1 (moves)   2:17:46

----------------------------------------------------------------------------------------
before:

Depths         MIDG END1 END2 END3     BOOK             TIME
Stockfish_11   17.1 17.5 22.9 25.2     3.5 (moves)     20:31
Stockfish_11   15.3 15.6 20.0 21.7     4.1 (moves)      0:00

Also still no crazy games and no suspect openings saved.

Shouldn't this be suspect by your definition with a margin of 1.00 or not?

[pgn][Event "Test_2020A"]
[Site "RWBC::CAPPUCCINO_Q8200"]
[Date "2020.02.18"]
[Round "448"]
[White "Stockfish_11-64"]
[Black "Stockfish_11x2-64"]
[Result "1-0"]
[BlackTimeControl "1/0.25"]
[ECO "A80"]
[Opening "Dutch"]
[PlyCount "94"]
[Termination "adjudication"]
[WhiteTimeControl "1/0.5"]

1. d4 {book} f5 {book}
2. Nc3 {book} d6 {book}
3. e4 {book} Nf6 {book}
4. exf5 {+1.36/17 1.92s} Bxf5 {-1.07/16 0.92s}
5. Nf3 {+1.41/14 0.41s} Qd7 {-1.02/13 0.10s}
6. Bc4 {+1.26/14 0.59s} Nc6 {-0.80/15 0.38s}
7. O-O {+1.64/16 0.50s} d5 {-0.92/14 0.27s}
8. Bb5 {+1.06/15 0.50s} Bg4 {-0.82/12 0.078s}
9. Re1 {+0.92/15 0.50s} e6 {-0.51/15 0.42s}
10. Qd3 {+0.83/16 0.50s} Bxf3 {-0.69/15 0.23s}
11. Qxf3 {+1.10/16 0.15s} Bd6 {-0.57/14 0.12s}
12. Qh3 {+0.68/18 0.84s} O-O {-0.51/15 0.17s}
13. Rxe6 {+0.95/18 0.27s} Rfe8 {-0.52/16 0.27s}
14. Rxe8+ {+0.77/18 0.57s} Rxe8 {-0.49/16 0.21s}
15. Be3 {+0.98/19 0.30s} Qxh3 {-0.30/17 0.27s}
16. gxh3 {+0.86/20 0.62s} a6 {-0.41/17 0.38s}
17. Bxc6 {+0.88/19 0.46s} bxc6 {-0.43/16 0.17s}
18. Rc1 {+0.85/17 0.41s} Nd7 {-0.57/17 0.44s}
19. Ne2 {+0.80/18 0.87s} c5 {-0.84/16 0.25s}
20. dxc5 {+0.84/15 0.15s} Bxc5 {-0.52/14 0.079s}
21. Bxc5 {+1.11/18 0.50s} Nxc5 {-0.88/18 0.42s}
22. Nf4 {+1.16/18 0.47s} c6 {-0.94/15 0.25s}
23. Nd3 {+1.00/18 0.88s} Nd7 {-0.75/13 0.14s}
24. c4 {+1.05/17 0.50s} Re2 {-0.53/16 0.36s}
25. cxd5 {+1.22/15 0.14s} cxd5 {-0.48/13 0.090s}
26. Kf1 {+1.72/18 0.86s} Re4 {-1.37/16 0.41s}
27. Rc8+ {+1.56/18 0.50s} Kf7 {-1.24/15 0.071s}
28. Rc7 {+1.52/18 0.16s} Ke8 {-1.31/19 0.23s}
29. Rxd7 {+1.49/20 0.84s} Kxd7 {-1.20/19 0.28s}
30. Nc5+ {+1.72/19 0.15s} Kd6 {-1.38/20 0.42s}
31. Nxe4+ {+1.65/21 0.50s} dxe4 {-1.27/16 0.086s}
32. h4 {+1.69/20 0.62s} g6 {-1.48/19 0.41s}
33. Ke2 {+2.06/21 0.38s} Ke5 {-1.65/18 0.21s}
34. Ke3 {+2.56/21 0.38s} a5 {-1.55/18 0.15s}
35. b3 {+3.11/20 0.39s} Kd5 {-2.61/19 0.39s}
36. a3 {+3.73/20 0.54s} h6 {-3.57/19 0.25s}
37. b4 {+4.84/22 0.48s} axb4 {-4.51/18 0.27s}
38. axb4 {+6.25/20 0.43s} h5 {-5.26/19 0.23s}
39. Kf4 {+12.32/23 0.46s} Kc4 {-3.12/18 0.17s}
40. Kg5 {+25.42/22 0.49s} Kxb4 {-13.17/19 0.33s}
41. Kxg6 {+61.52/30 0.56s} Kc5 {-60.39/27 0.25s}
42. Kxh5 {+72.43/32 0.45s} Kd4 {-60.56/26 0.25s}
43. Kg6 {+M31/31 1.16s} Kd3 {-70.93/28 0.23s}
44. Kf5 {+M27/31 0.36s} Kd2 {-72.07/24 0.27s}
45. Kxe4 {+M23/32 0.20s} Kc3 {-M26/27 0.25s}
46. h5 {+M21/33 0.36s} Kc4 {-M22/27 0.089s}
47. h6 {+M19/33 0.44s} Kb3 {-M20/28 0.21s}
{White wins by adjudication}
1-0[/pgn]

[Rebel]

I am still not sure about point 3., shouldn't it report then a 'suspect openings', if e.g. eval > 1.0 for White and White wins?
Or does it consider the eval of the reply too?
This would be a big improvement anyway. What, if you just record it as 'suspect', if the eval is >X for a few moves (let's say 3 or 4 again)?

I hear you, but it's complicated. If you look at the game black starts with a -1.07 score but during the game the score increases close to a draw score and following the code the opening is not suspect. Following your suggestion the game wouldn't make it to suspect status either Lemme think about it.

[Rebel]

After the change in the pgn I got at least two different time results compared to before.
But still something there seems to be wrong? The test lasted over 16:03:xx.

16 minutes or 16 hours ?

Code: Select all

after adding (s) again to move times:

Depths         MIDG END1 END2 END3     BOOK             TIME
Stockfish_11   17.1 17.5 22.9 25.2     3.5 (moves)   5:09:40
Stockfish_11   15.3 15.6 20.0 21.7     4.1 (moves)   2:17:46

----------------------------------------------------------------------------------------
before:

Depths         MIDG END1 END2 END3     BOOK             TIME
Stockfish_11   17.1 17.5 22.9 25.2     3.5 (moves)     20:31
Stockfish_11   15.3 15.6 20.0 21.7     4.1 (moves)      0:00

Note that you need to divide time by the number of cores you used.

But I noticed a small bug anyway, MRI stops counting time after 190 moves, typo.

Also still no crazy games and no suspect openings saved.

Crazy games only occur when 2 engines firmly disagree with each other and both show a possitive score > margin.

Shouldn't this be suspect by your definition with a margin of 1.00 or not?

[pgn][Event "Test_2020A"]
[Site "RWBC::CAPPUCCINO_Q8200"]
[Date "2020.02.18"]
[Round "448"]
[White "Stockfish_11-64"]
[Black "Stockfish_11x2-64"]
[Result "1-0"]
[BlackTimeControl "1/0.25"]
[ECO "A80"]
[Opening "Dutch"]
[PlyCount "94"]
[Termination "adjudication"]
[WhiteTimeControl "1/0.5"]

1. d4 {book} f5 {book}
2. Nc3 {book} d6 {book}
3. e4 {book} Nf6 {book}
4. exf5 {+1.36/17 1.92s} Bxf5 {-1.07/16 0.92s}
5. Nf3 {+1.41/14 0.41s} Qd7 {-1.02/13 0.10s}
6. Bc4 {+1.26/14 0.59s} Nc6 {-0.80/15 0.38s}
7. O-O {+1.64/16 0.50s} d5 {-0.92/14 0.27s}
8. Bb5 {+1.06/15 0.50s} Bg4 {-0.82/12 0.078s}
9. Re1 {+0.92/15 0.50s} e6 {-0.51/15 0.42s}
10. Qd3 {+0.83/16 0.50s} Bxf3 {-0.69/15 0.23s}
11. Qxf3 {+1.10/16 0.15s} Bd6 {-0.57/14 0.12s}
12. Qh3 {+0.68/18 0.84s} O-O {-0.51/15 0.17s}
13. Rxe6 {+0.95/18 0.27s} Rfe8 {-0.52/16 0.27s}
14. Rxe8+ {+0.77/18 0.57s} Rxe8 {-0.49/16 0.21s}
15. Be3 {+0.98/19 0.30s} Qxh3 {-0.30/17 0.27s}
16. gxh3 {+0.86/20 0.62s} a6 {-0.41/17 0.38s}
17. Bxc6 {+0.88/19 0.46s} bxc6 {-0.43/16 0.17s}
18. Rc1 {+0.85/17 0.41s} Nd7 {-0.57/17 0.44s}
19. Ne2 {+0.80/18 0.87s} c5 {-0.84/16 0.25s}
20. dxc5 {+0.84/15 0.15s} Bxc5 {-0.52/14 0.079s}
21. Bxc5 {+1.11/18 0.50s} Nxc5 {-0.88/18 0.42s}
22. Nf4 {+1.16/18 0.47s} c6 {-0.94/15 0.25s}
23. Nd3 {+1.00/18 0.88s} Nd7 {-0.75/13 0.14s}
24. c4 {+1.05/17 0.50s} Re2 {-0.53/16 0.36s}
25. cxd5 {+1.22/15 0.14s} cxd5 {-0.48/13 0.090s}
26. Kf1 {+1.72/18 0.86s} Re4 {-1.37/16 0.41s}
27. Rc8+ {+1.56/18 0.50s} Kf7 {-1.24/15 0.071s}
28. Rc7 {+1.52/18 0.16s} Ke8 {-1.31/19 0.23s}
29. Rxd7 {+1.49/20 0.84s} Kxd7 {-1.20/19 0.28s}
30. Nc5+ {+1.72/19 0.15s} Kd6 {-1.38/20 0.42s}
31. Nxe4+ {+1.65/21 0.50s} dxe4 {-1.27/16 0.086s}
32. h4 {+1.69/20 0.62s} g6 {-1.48/19 0.41s}
33. Ke2 {+2.06/21 0.38s} Ke5 {-1.65/18 0.21s}
34. Ke3 {+2.56/21 0.38s} a5 {-1.55/18 0.15s}
35. b3 {+3.11/20 0.39s} Kd5 {-2.61/19 0.39s}
36. a3 {+3.73/20 0.54s} h6 {-3.57/19 0.25s}
37. b4 {+4.84/22 0.48s} axb4 {-4.51/18 0.27s}
38. axb4 {+6.25/20 0.43s} h5 {-5.26/19 0.23s}
39. Kf4 {+12.32/23 0.46s} Kc4 {-3.12/18 0.17s}
40. Kg5 {+25.42/22 0.49s} Kxb4 {-13.17/19 0.33s}
41. Kxg6 {+61.52/30 0.56s} Kc5 {-60.39/27 0.25s}
42. Kxh5 {+72.43/32 0.45s} Kd4 {-60.56/26 0.25s}
43. Kg6 {+M31/31 1.16s} Kd3 {-70.93/28 0.23s}
44. Kf5 {+M27/31 0.36s} Kd2 {-72.07/24 0.27s}
45. Kxe4 {+M23/32 0.20s} Kc3 {-M26/27 0.25s}
46. h5 {+M21/33 0.36s} Kc4 {-M22/27 0.089s}
47. h6 {+M19/33 0.44s} Kb3 {-M20/28 0.21s}
{White wins by adjudication}
1-0[/pgn]

See previous post.

[Rebel]

One more thing, I noticed your Cutechess PGN has the time in 2 decimals (0.92s), mine has only 1 decimal (0.9s).

[Guenther]

One more thing, I noticed your Cutechess PGN has the time in 2 decimals (0.92s), mine has only 1 decimal (0.9s).

Yes, I wrote this already in my first post. I compiled Cutechess and CuteChess for doing so, because with short tc stats, I need more precision.
Other people use that compilation too BTW. This will become a future improvement in cutechess/cli anyway.

BTW even the default cutechess can have up to 3 digits precision in move times for very low times.
https://github.com/cutechess/cutechess/ ... me.cpp#L42

[Guenther]

I am still not sure about point 3., shouldn't it report then a 'suspect openings', if e.g. eval > 1.0 for White and White wins?
Or does it consider the eval of the reply too?
This would be a big improvement anyway. What, if you just record it as 'suspect', if the eval is >X for a few moves (let's say 3 or 4 again)?

I hear you, but it's complicated. If you look at the game black starts with a -1.07 score but during the game the score increases close to a draw score and following the code the opening is not suspect. Following your suggestion the game wouldn't make it to suspect status either :) Lemme think about it.

Well, that's why I asked what exactly you are doing for recognizing a suspect opening.
If you don't tell, I cannot check if something is wrong.