Minic 3.18 NN endgame problem ... (14 games, *.zip)

[Frank Quisinsky]

Hi there,

14 games of 1.200 games, Minic 3.18 NN lost points (draw games, mate a long time before possible).
Such problems can be see in games without resign-mode!

7 points is a lot.
Minic lost around 5 Elo.

https://www.amateurschach.de/download/m ... roblem.zip

Best
Frank

[xr_a_y]

Thanks a lot for reporting. I'll study that.

[Frank Quisinsky]

Hi Vivien,

I like your program a lot, problem should be fixed (underpromotions).
Sorry for my open message, have no email address in outlook.
Not sure that you find the message in thread about bug reports!

Go Minic go ...

A small wish ...
Maybe you have the bugfix ready to start for my run-4 (middle of April, 2022).

Best
Frank

[xr_a_y]

If the bug fix about underpromotion is the one where promotes to N (respectively B) in endgame with B (respectively N) to end up with BN instead of NQ or BQ, then it is not a bug, it's a feature

But when Minic fail to win an endgame, this is real bug. I really though this one does not exist anymore but it seems to still be here somehow...

I'm looking at it right now.

Next release will be a massive Elo jump for no increment sudden death TC like 1min+0sec, but a neural release for any other TC.

[xr_a_y]

The second game in your zip file is not that horrible.
Minic is simplifying to a drawn endgame instead of keeping the rooks on the board but no big issue here I think

[pgn]
[Event "40 Moves in 20 min"]
[Site "FCP Tourney-KI, FQ-3"]
[Date "2022.02.28"]
[Round "?"]
[White "Clover 3.0 NN"]
[Black "Minic 3.18 NN"]
[Result "1/2-1/2"]
[ECO "C80"]
[Opening "Ruy Lopez: open (7.Bb3)"]

1. e4 {book 0s} 1... e5 {book 0s} 2. Nf3 {book 0s} 2... Nc6 {book 0s} 3. Bb5
{book 0s} 3... Nf6 {book 0s} 4. O-O {book 0s} 4... Nxe4 {book 0s} 5. d4 {book
0s} 5... a6 {book 0s} 6. Ba4 {book 0s} 6... b5 {book 0s} 7. Bb3 {book 0s}
7... Be7 {book 0s} 8. Nxe5 {book 0s} 8... Nxe5 {book 0s} 9. dxe5 {book 0s}
9... Bb7 {book 0s} 10. Qg4 {book 0s} 10... O-O {book 0s} 11. f3 {book 0s}
11... Ng5 {book 0s} 12. f4 {book 0s} 12... Ne4 {book 0s} 13. Be3 {-0.14/23 21s}
13... c5 {-0.27/27 23s} 14. c3 {-0.59/27 26s} 14... c4 {-0.54/29 26s} 15. Bc2
{-0.13/26 59s} 15... f5 {-0.54/30 20s} 16. Qh3 {+0.01/25 23s (Qe2)} 16... a5
{-0.65/27 26s} 17. Bxe4 {-0.91/26 1:06m} 17... Bxe4 {-0.86/28 23s} 18. Nd2
{-0.76/24 26s} 18... Bd3 {-0.61/28 29s} 19. Rfc1 {-0.69/27 43s (Rfe1)} 19... d6
{-1.11/26 42s} 20. Nf3 {-0.96/27 47s} 20... Be4 {-1.08/25 26s} 21. Rd1 {-0.94/24
44s} 21... Ra6 {-0.90/27 25s} 22. a3 {-0.94/25 26s (Qh5)} 22... h6 {-0.87/26
26s} 23. Rd2 {-1.09/26 1:19m} 23... Qc8 {-0.84/27 28s} 24. Bd4 {-0.85/28 45s
(Qg3)} 24... Qb7 {-0.78/27 28s} 25. Qh5 {-0.93/27 52s (Qg3)} 25... Kh7 {-1.06/29
34s} 26. Rad1 {-0.79/25 25s} 26... d5 {-1.17/32 39s} 27. Rf2 {-0.92/28 28s}
27... b4 {-1.42/31 31s} 28. axb4 {-0.94/26 21s} 28... axb4 {-1.59/31 44s}
29. Ng5+ {-1.06/28 29s} 29... Bxg5 {-1.40/32 50s} 30. fxg5 {-0.96/30 29s}
30... b3 {-1.64/31 47s} 31. gxh6 {-0.67/30 33s} 31... gxh6 {-1.28/31 31s}
32. Qh4 {-0.71/27 27s} 32... Qg7 {-1.56/33 32s} 33. e6 {-1.02/29 26s} 33... Qg5
{-1.49/34 57s} 34. Qxg5 {-1.22/30 32s (e7)} 34... hxg5 {-1.91/33 59s} 35. e7
{-0.90/29 31s} 35... Re8 {-1.69/32 1:01m} 36. Bc5 {-1.06/31 34s} 36... g4
{-1.91/31 40s} 37. Rfd2 {-1.26/31 58s} 37... Ra2 {-2.17/31 50s} 38. Re1
{-1.10/29 32s (Kf2)} 38... Kg6 {-2.30/31 48s} 39. Kf2 {-1.42/32 1:08m} 39... f4
{-2.10/34 1:39m} 40. Bb4 {-1.49/29 42s} 40... Rh8 {-2.26/32 1:34m} 41. Bd6
{-1.49/29 41s (Kg1)} 41... Kf5 {-2.30/32 17s} 42. Bc5 {-1.49/30 28s (Kg1)}
42... Ra5 {-2.52/30 18s} 43. Bb4 {-1.58/28 20s} 43... Raa8 {-2.58/32 26s}
44. Kg1 {-1.44/28 35s} 44... Rh6 {-2.41/31 27s} 45. Bc5 {-1.39/30 57s (Rf2)}
45... Rc6 {-2.34/32 27s} 46. Bb4 {-1.39/32 20s} 46... Re6 {-2.34/30 23s} 47. Bc5
{-1.39/32 18s} 47... Re8 {-2.40/31 27s} 48. Red1 {-1.47/32 42s (Bb4)} 48... Bd3
{-2.28/32 27s} 49. Kf2 {-1.38/28 17s} 49... Rc6 {-1.90/30 28s} 50. Bb4 {-1.48/31
38s} 50... Ke5 {-1.91/30 21s} 51. Kg1 {-1.45/30 1:05m (Re1+)} 51... Ra6
{-2.37/29 22s} 52. Bc5 {-1.57/28 43s (Re1+)} 52... Ra5 {-2.35/29 29s} 53. Re1+
{-1.39/31 27s (Bb6)} 53... Be4 {-2.37/28 20s} 54. Bb4 {-1.57/33 31s} 54... Ra6
{-2.18/29 21s} 55. Bc5 {-1.56/32 35s (Rf1)} 55... Rg8 {-2.38/30 33s} 56. Rf1
{-1.56/30 40s} 56... Kf5 {-2.32/30 19s} 57. Rfd1 {-1.57/30 18s (Re1)} 57... Ke6
{-2.37/29 17s} 58. Kf2 {-1.56/31 19s (Rf2)} 58... Ra5 {-2.85/27 28s} 59. Bb4
{-1.57/28 27s} 59... Raa8 {-2.51/33 30s} 60. Kg1 {-1.51/30 27s (Re1)} 60... g3
{-2.33/28 25s} 61. hxg3 {-1.89/29 37s} 61... Rxg3 {-2.39/31 41s} 62. Kf1
{-1.61/33 23s} 62... Kf7 {-2.51/31 29s} 63. Rf2 {-2.56/30 40s (Bd6)} 63... f3
{-2.57/34 29s} 64. Rxf3+ {-2.83/31 18s (gxf3)} 64... Bxf3 {-2.58/38 19s}
65. gxf3 {-2.93/32 24s} 65... Rxf3+ {-3.80/41 23s} 66. Ke2 {-3.39/31 37s}
66... Rf5 {-3.04/34 20s} 67. Rh1 {-2.52/29 59s} 67... Rg8 {-3.05/38 19s}
68. Rh7+ {-3.54/33 17s} 68... Ke6 {-2.94/37 22s} 69. Rh6+ {-3.54/33 18s}
69... Rf6 {-3.29/38 36s} 70. Rxf6+ {-0.56/33 19s (Rh5)} 70... Kxf6 {-0.73/45
25s} 71. Ke3 {-0.01/37 19s} 71... Ke5 {-0.23/49 44s} 72. Ba3 {-0.01/50 51s
(Bc5)} 72... Rh8 {-0.01/48 21s} 73. Bb4 {-0.01/49 19s} 73... Rc8 {-0.01/50 53s}
74. Bc5 {-0.01/43 25s} 74... Ke6 {-0.01/49 21s} 75. Kd4 {-1.69/46 26s} 75... Rg8
{0.00/48 26s} 76. Ba3 {-0.01/52 42s} 76... Rg4+ {0.00/49 27s} 77. Kc5 {-0.01/52
23s} 77... Kxe7 {0.00/48 44s} 78. Kxd5+ {0.00/54 21s} 78... Kd7 {0.00/49 34s}
79. Bb4 {0.00/62 47s (Bc5)} 79... Kc8 {0.00/48 48s} 80. Bc5 {0.00/60 20s}
80... Rg2 {0.00/53 1:07m} 81. Kxc4 {0.00/69 15s} 81... Rxb2 {0.00/54 13s}
82. Kb4 {0.00/68 18s (Bb4)} 82... Kd8 {0.00/56 13s} 83. Bb6+ {0.00/68 17s (Bd4)}
83... Kd7 {0.00/54 13s} 84. Ka3 {0.00/66 17s (c4)} 84... Kc6 {0.00/59 19s}
85. Bd4 {0.00/70 16s (Kxb2)} 85... Rh2 {0.00/60 19s} 86. Kxb3 {0.00/141 18s}
86... Rh3 {0.00/36 19s} 87. Be5 {book 0s (Bg7)} 87... Rf3 {0.00/36 17s} 88. Kb4
{book 0s} 88... Rd3 {0.00/34 16s} 89. Bd4 {book 0s} 89... Rd1 {0.00/34 20s}
90. Bf6 {book 0s (Kc4)} 90... Rf1 {0.00/39 18s} 91. Bd8 {book 0s (Bg7)}
91... Kd7 {0.00/36 18s} 92. Bb6 {book 0s (Ba5)} 92... Kd6 {0.00/36 17s} 93. Be3
{book 0s (Bc5+)} 93... Kd5 {0.00/44 19s} 94. Bd2 {book 0s (c4+)} 94... Rb1+
{-0.01/45 20s} 95. Ka5 {book 0s} 95... Kc4 {0.00/40 20s} 96. Be3 {book 0s (Bf4)}
96... Kxc3 {0.00/62 19s} 97. Bg5 {book 0s} 97... Kc4 {-0.13/36 20s} 98. Bf6
{book 0s (Ka6)} 98... Rb5+ {-30.81/53 17s} 99. Ka6 {book 0s} 99... Kc5
{-30.81/55 17s} 100. Bd4+ {book 0s (Ka7)} 100... Kc6 {-28.73/57 27s} 101. Ka7
{book 0s} 101... Rb3 {-28.73/57 18s} 102. Bf2 {book 0s (Bg1)} 102... Rb1
{-28.72/57 23s} 103. Bg3 {book 0s (Be3)} 103... Rb7+ {-21.20/58 20s} 104. Ka8
{book 0s} 104... Rf7 {-21.14/58 27s} 105. Bh2 {book 0s (Kb8)} 105... Kb6
{-20.84/55 21s} 106. Be5 {book 0s (Bg1+)} 106... Rf8+ {-20.00/55 32s} 107. Bb8
{book 0s} 107... Rf3 {-19.91/58 25s} 108. Bd6 {book 0s (Ba7+)} 108... Kc6
{-18.74/59 25s} 109. Be5 {book 0s} 109... Rf8+ {-18.54/60 25s} 110. Ka7 {book
0s} 110... Rf5 {-17.42/59 30s} 111. Bg3 {book 0s} 111... Rf1 {-12.37/57 40s}
112. Bh2 {book 0s} 112... Rf2 {-11.25/59 28s} 113. Be5 {book 0s} 113... Ra2+
{-8.42/60 36s} 114. Kb8 {book 0s} 114... Ra5 {-11.15/57 39s} 115. Bg3 {book 0s
(Bf4)} 115... Ra3 {-39.92/56 42s} 116. Bh2 {book 0s (Be5)} 116... Rh3 {-20.32/56
38s} 117. Be5 {book 0s (Bf4)} 117... Rb3+ {-35.38/56 1:36m} 118. Ka7 {book 0s}
118... Rb7+ {-12.42/51 1:01m} 119. Ka8 {book 0s} 119... Re7 {-4.31/50 1:25m}
120. Bf6 {book 0s} 120... Re3 {-37.20/49 2:19m} 121. Bh4 {book 0s (Bd4)}
121... Rf3 {-1.87/45 31s} 122. Bg5 {book 0s (Ka7)} 122... Kc7 {-15.98/42 23s}
123. Bc1 {book 0s} 123... Kb6 {-19.20/39 33s} 124. Bh6 {book 0s} 124... Rg3
{-3.73/39 51s} 125. Bf4 {book 0s} 125... Rg8+ {-1.82/35 33s} 126. Bb8 {book 0s}
126... Rg2 {-2.24/35 21s} 127. Bd6 {book 0s (Be5)} 127... Re2 {-4.01/34 17s}
128. Bg3 {book 0s (Bf4)} 128... Re8+ {-16.82/30 27s} 129. Bb8 {book 0s}
129... Kc6 {-8.09/26 28s} 130. Ka7 {book 0s} 130... Re7+ {-35.71/28 34s}
131. Ka8 {book 0s} 131... Kb6 {-27.05/27 30s} 132. Bf4 {book 0s} 132... Ra7+
{-21.89/27 21s} 133. Kb8 {book 0s} 133... Rf7 {-36.22/25 16s} 134. Be3+ {book
0s} 134... Kc6 {-36.24/26 23s} 135. Bg5 {book 0s (Ka8)} 135... Kc5 {-36.74/31
2:41m} 136. Bc1 {book 0s (Be3+)} 136... Kc4 {-1.96/23 38s} 137. Bb2 {book 0s
(Bd2)} 137... Kd5 {-0.01/35 1:51m} 138. Ba1 {book 0s (Kc8)} 138... Rf1 {-0.01/37
2:02m} 139. Bc3 {book 0s} 139... Kc6 {-0.01/28 9s} 140. Bd2 {book 0s (Be5)}
140... Rf8+ {-0.01/41 47s} 141. Ka7 {book 0s} 141... Kc7 {-0.01/31 9s} 142. Be1
{book 0s (Bc1)} 142... Kc6 {-0.01/42 9s} 143. Bg3 {book 0s (Bd2)} 143... Kb5
{-0.01/83 7s} 144. Bh2 {book 0s} 144... Ka4 {-0.01/121 2s} 145. Be5 {book 0s}
145... Rf5 {-0.01/121 0s} 146. Bd6 {book 0s (Bc3)} 146... Kb3 {+0.01/121 0s}
1/2-1/2
[/pgn]

Anyway I'm really surprised those end-games are not well played. Any chance you activate "forceNNUE" ? this would use the network for every position and shall not be activated for engine/engine matches.
Anyway, I'll have a deeper look into that, it is indeed really not pretty.

Thanks again for reporting that.

[xr_a_y]

I have generated using a python script around 20000 KNBvK endgame positions and run a match from here SF versus Minic at STC (1s+0.1) to evaluate this probability of failure (here to draw). On this specific end-game it is quite a lot (around 11% of the games that shall be won here)

Code: Select all

Score of minic_dev_dev vs stockfish-dev: 21450 - 24425 - 2686 [0.469]
...      minic_dev_dev playing White: 10692 - 12216 - 1373  [0.469] 24281
...      minic_dev_dev playing Black: 10758 - 12209 - 1313  [0.470] 24280
...      White vs Black: 22901 - 22974 - 2686  [0.499] 48561
Elo difference: -21.3 +/- 3.0, LOS: 0.0 %, DrawRatio: 5.5 %

and looses some 20Elo.

I'll analyze those games today if I have time

[Frank Quisinsky]

Hi Vivien,

at the moment Minic 3.18 NN is still playing the "run-3".
1.200 games for "run-2"
540 new games for "run-3".

During this time "run-3" is still running I will not make any changes in the configuration.

A comparable "endgame-problem" have Bit-Genie 9.19 dev.
For "run-3" I replaced Bit-Genie 9.19 dev.
Bit-Genie have much other problems in loosing games to fast.
Different problems are in the program.

Back to Minic:

To the KB-KR position.
I added only one game in ".zip file, much more are in the database of 1.200 games.
Here the eval isn't correct, nothing to do with lost 0.5 points.

Underpromotion:
Differnet siutaitons:
K and 2x pawn vs. K ... I believe 2-3 times.

I am looking in one of the games during still running.
I am thinking ... mate in 2 with promotion to queen ... nice the next game can be start.

But Minic made the first underpromotion in knight and later the second underpromotion in bishop and can't set mate.
After mate in 2 is possible the game ended 70 minutes later with draw.

Thinking on persons search in the databases of high quality games (I am playing here 80 minutes games) for underpromotions. If I gave with the tool "joined" the information ... in database are x underpromotions to knights, x to bishops and x to rooks. People like to search for underpromotions and stalemate positions.



Underpromotion is nice but no reason to make an underpromotion if games runs to long.
My game statistics to move-average will be bad also!

In some other cases Minic 3.18 NN can't mate with KR vs. K.

Maybe "forceNNUE" should be standard as setting?

I am very happy that you are know looking in detail.
Thinking on a discussion for around 1-2 years with your person.
The problem that Minic produced crashes in endgames with Shredder GUI is solved.
In all the games with Minic 3.18 NN no more problems with Shredder GUI.

All your work around your own Neural Network is great. The work on Minic is great.
A very strong and interesting engine you produced.
But for looking in live mode ... if an endgame is in the near it is not nice.

Good that you are working on it !!

If you have a bugfix for run-4 ...
Minic 3.x NN will play again and will be much stronger.

Best
Frank

You know:
So many engines have problems in endgames.

Komodo is a good example:
Do you know how many time (80-minutes games) I lost here if Komodo with 4 or 5 pieces on board thinking for each move a lot of seconds / minutes with syzygybases. Often I am thinking different engines have interst to find out that 4-men and 5-men endgame databases are wrong. The moves must be play directly. And so many other things I dislike here.

[Frank Quisinsky]

ForceNNUE ...

If in games +5 - +8 (for an example) no reason to play with an NNUE file.
In case of -5 - -8 NNUE can help.

Own eval is more intersting and engine is faster without NN file (speed factor is here more interesting as the NNUE file).
Without NNUE engine is faster ...

Best
Frank

I wish me that more programmers looking in the endgames for solve easy problems.
All the games I produced are allways without resign-mode.

[Modern Times]

Vivien has previously said that ForceENNUE weakens the engine and should not be selected for engine matches. The default is off, which means it uses HCE in some situations and NNUE in others.

[xr_a_y]

Vivien has previously said that ForceENNUE weakens the engine and should not be selected for engine matches. The default is off, which means it uses HCE in some situations and NNUE in others.

Indeed.