Really serious endgame issues

[jdart]

Null-move pruning can cause blindness in certain endgames. I scale back that type of pruning in low-material endgames.

[abulmo2]

Same for OliThink and Dumb, even 137 ELO:

Code: Select all

   # PLAYER             :  RATING  ERROR  POINTS  PLAYED   (%)     W    D     L  D(%)  CFS(%)
   1 OliThink 5.11.4    :       0   ----  2087.0    3047  68.5  1779  616   652  20.2     100
   2 Dumb 2.3           :    -137     12   960.0    3047  31.5   652  616  1779  20.2     ---

I tested Dumb 2.3 against OliThink 5.10.5, and I confirm Dumb was already significantly weaker than Olithink 5.10.5 by 68 Elo (+/-1). I can imagine Olithink 5.11.4 to be another 70 Elo stronger.

Code: Select all

# PLAYER             : RATING  ERROR   POINTS  PLAYED    (%)
   1 OliThink 5.10.5    : 2334.0    1.3  51348.5   86280   59.5%
   2 Dumb 2.3            : 2266.0    1.3  34931.5   86280   40.5%

In those tournaments Dumb and Qapla perform better than Olithink. I have seen it[ live. Example above.
What exactly is bringing the performance of OliThink so down in those tournaments? Can it be just the CPU? Smaller Level 1 Cache? But 130 ELO??

Those tournaments have a small number of games, so bad luck is to be considered in the result of the games.
I use the same CPU as Graham Banks. What astonished me is the low nps showed by Dumb. On my computer, a Ryzen 9 5950x running at 4.2 Ghz. Dumb is twice faster than in those tournaments, even when running under Windows 11 with ChessGUI. I wonder if there is something wrong in Graham Banks' setup. Maybe some engine are affected differently which may explain some strange results.

[OliverBr]

Those tournaments have a small number of games, so bad luck is to be considered in the result of the games.

Not exactly, because there a at least 50 tournaments of Graham where OliThink does constantly underperfom to anything I test myself. OliThink was relegated about 6 times, where other engines, always weaker in my tests, were not.
OliThink's sparring partner are about 5 divisions higher.

I use the same CPU as Graham Banks. What astonished me is the low nps showed by Dumb. On my computer, a Ryzen 9 5950x running at 4.2 Ghz. Dumb is twice faster than in those tournaments, even when running under Windows 11 with ChessGUI. I wonder if there is something wrong in Graham Banks' setup. Maybe some engine are affected differently which may explain some strange results.

This may be an answer because it's true that the difference between OliThink and other engines melts with increasing time control. And if there is something slowing down it could affect OliThink negatively.

I have watched a couple of games on the last tournament (118th, 11. Division) and every game that was lost, was lost in the endgame. OliThink entered the endgame with drawish evaluation and lost.
OliThink has an evaluation based on mobility. And it could be that this evaluation is not just enough in an endgame. Especially when it's "slow". This is just an guess.

[OliverBr]

Null-move pruning can cause blindness in certain endgames. I scale back that type of pruning in low-material endgames.

Thank you. Have been thinking on that, too. But actually OliThink is stricter than most games with null move on endgamesm because the following condition:

Code: Select all

(bitcnt(P.color[c] & (~P.piece[PAWN]) & (~pinnedPieces(P.king[c], oc))) > 1) 

requests, that there a at least 2 non-pawn pieces and that they are not pinned. That is something I never saw anywhere else, was just my idea.

[Graham Banks]

Those tournaments have a small number of games, so bad luck is to be considered in the result of the games.

Not exactly, because there a at least 50 tournaments of Graham where OliThink does constantly underperfom to anything I test myself. OliThink was relegated about 6 times, where other engines, always weaker in my tests, were not.
OliThink's sparring partner are about 5 divisions higher.

I use the same CPU as Graham Banks. What astonished me is the low nps showed by Dumb. On my computer, a Ryzen 9 5950x running at 4.2 Ghz. Dumb is twice faster than in those tournaments, even when running under Windows 11 with ChessGUI. I wonder if there is something wrong in Graham Banks' setup. Maybe some engine are affected differently which may explain some strange results.

This may be an answer because it's true that the difference between OliThink and other engines melts with increasing time control. And if there is something slowing down it could affect OliThink negatively.

I have watched a couple of games on the last tournament (118th, 11. Division) and every game that was lost, was lost in the endgame. OliThink entered the endgame with drawish evaluation and lost.
OliThink has an evaluation based on mobility. And it could be that this evaluation is not just enough in an endgame. Especially when it's "slow". This is just an guess.

How do you have OliThink set up?
I have it installed as a Winboard engine (Windows), but with no ini file I cannot change any parameters unless there are command line ones I can give the GUI.

I can open OliThink and give it a depth search if you'd like to compare the output with what you get.

I can do the same for Dumb if Richard wishes.

[OliverBr]

How do you have OliThink set up?

It is set up as it comes out of the box. I can read and file "olithink.pgn", if there is one for some openings.
Everything else is the Windows (CECP) Protocol which it understands once startet.

I have it installed as a Winboard engine (Windows), but with no ini file I cannot change any parameters unless there are command line ones I can give the GUI.

There is actually one command line command I am often using: "-sd", which means "Search depth", e.g.:

Code: Select all

olithink.exe -sd 25

followed by

Code: Select all

e3

The e3 is Winboard protocol command and means: "Move e3!". Why e3? Because there is no opening in the small book with "1. e3".
The output für OliThink 5.11.4 is:

Code: Select all

  1   -47      0         5  a7a6 
 2   -58      0        24  a7a6 b2b3 
 3    -3      0        86  d7d5 b2b3 c7c5 
 4    17      0       351  d7d5 a2a4 e7e5 d2d4 
 5    19      0       763  d7d5 d2d4 b8c6 h2h3 e7e6 
 6    12      0      1298  d7d5 d2d4 b8c6 h2h3 e7e5 c2c3 
 7     6      0      2359  d7d5 d2d4 g8f6 f2f3 e7e6 c2c3 
 8   -11      0      7356  d7d5 d2d4 g8f6 b1c3 c8f5 f1d3 e7e6 d3f5 
 9     0      1     13860  d7d5 d2d4 e7e6 b1c3 b8c6 c1d2 g8f6 g1f3 c8d7 
10   -14      2     45761  e7e5 d2d4 e5e4 d4d5 g8f6 b1c3 f8d6 c1d2 
11     0      4    103082  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 b8c6 f1d3 f8d6 e1g1 e8g8 
12   -12      5    133079  d7d5 d2d4 g8f6 g1f3 e7e6 f3e5 b8c6 e5c6 b7c6 b1c3 f6e4 c3e4 
13    -2      8    194120  d7d5 d2d4 g8f6 g1f3 e7e6 f1d3 b8c6 d1e2 f8d6 b1c3 e8g8 e1g1 
14    -7     13    326344  e7e5 d2d4 e5e4 d4d5 g8f6 b1c3 f8d6 f1e2 e8g8 g1h3 f8e8 e1g1 c7c6 
15   -10     20    601562  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 f8d6 f1d3 a7a6 e1g1 e8g8 c1d2 c8d7 f3e5 b8c6 
16   -10     26    805368  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 f8d6 f1d3 a7a6 e1g1 c8d7 c1d2 e8g8 f3e5 b8c6 
17    -9     39   1282884  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 c8d7 f1e2 f8d6 f3e5 b8c6 e5d7 d8d7 e1g1 e8g8 c1d2 f8e8 
18    -9     55   1873354  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 c8d7 f1e2 f8d6 f3e5 b8c6 e5d7 d8d7 e1g1 e8g8 c1d2 f8e8 
19   -12     85   2965790  d7d5 d2d4 g8f6 g1f3 e7e6 f1e2 f8d6 c2c4 d5c4 e2c4 e8g8 b1c3 b8d7 e1g1 d7b6 c4d3 c8d7 c1d2 d7c6 f3e5 
20   -12    130   4593646  d7d5 d2d4 g8f6 g1f3 e7e6 f1e2 f8d6 c2c4 d5c4 e2c4 e8g8 b1c3 b8d7 e1g1 d7b6 c4d3 c8d7 c1d2 d7c6 f3e5 
21   -12    179   6396786  d7d5 d2d4 g8f6 g1f3 e7e6 f1e2 c7c5 b1c3 f8e7 e1g1 e8g8 c1d2 b8c6 f3e5 c8d7 a2a3 c5c4 e5d7 d8d7 c3b5 
22   -12    341  12560946  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 c1d2 b8c6 f1e2 f6e4 e1g1 e8g8 d1b3 a7a6 f1c1 e4d2 b1d2 c6a5 b3c2 a5c6 
23    -8    454  16747642  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 b1c3 e8g8 f1e2 b8c6 e1g1 f6e4 c1d2 a7a6 g1h1 g8h8 d1e1 f7f5 c4d5 e4d2 e1d2 e6d5 
24    -8    603  21205553  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 b1c3 e8g8 f1e2 b8c6 e1g1 f6e4 c1d2 a7a6 g1h1 g8h8 d1e1 f7f5 c4d5 e4d2 e1d2 e6d5 
25    -9    800  27120003  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 c4d5 e6d5 f1d3 e8g8 b1c3 b8a6 a2a3 c7c5 d3c2 c5c4 e1g1 d8b6 c3a4 b6e6 c1d2 f8d8 
move d7d5

So this would be very interesting on the very computer it is playing the tournaments and then same again for OliThink 5.11.5, because I made changes which affects CPU Cache.

I can open OliThink and give it a depth search if you'd like to compare the output with what you get.

Thank you!
Alternatively you could open you GUI with OliThink with enough time and make die "e3" there.
OliThink can also "analyse" mode but this should be startet after you made "e3" because else it's starting to calculate before already and the output will be different.

I can do the same for Dumb if Richard wishes.

I have said before that I like "Dumb" a lot as a engine, because it is genuine.

[Graham Banks]

I use a generic opening book for all engines in a tournament.

[OliverBr]

I tested Dumb 2.3 against OliThink 5.10.5, and I confirm Dumb was already significantly weaker than Olithink 5.10.5 by 68 Elo (+/-1). I can imagine Olithink 5.11.4 to be another 70 Elo stronger.

Actually 5.11.4 is supposed to be about 30 Elo stronger than 5.10.5, but this is always some kind of surprise bag regarding time controls. I cannot test thousands of games with 40/120 or slower, because this needs days.
In a test with Qapla the the lead literally melted with slower time controls. I had this years before with Leela (Lc0) who was as strong as Stockfish, but OliThink was better in fast time controls. (Not compared to stockfish, lol, never)

[Graham Banks]

How do you have OliThink set up?

It is set up as it comes out of the box. I can read and file "olithink.pgn", if there is one for some openings.
Everything else is the Windows (CECP) Protocol which it understands once startet.

I have it installed as a Winboard engine (Windows), but with no ini file I cannot change any parameters unless there are command line ones I can give the GUI.

There is actually one command line command I am often using: "-sd", which means "Search depth", e.g.:

Code: Select all

olithink.exe -sd 25

followed by

Code: Select all

e3

The e3 is Winboard protocol command and means: "Move e3!". Why e3? Because there is no opening in the small book with "1. e3".
The output für OliThink 5.11.4 is:

Code: Select all

  1   -47      0         5  a7a6 
 2   -58      0        24  a7a6 b2b3 
 3    -3      0        86  d7d5 b2b3 c7c5 
 4    17      0       351  d7d5 a2a4 e7e5 d2d4 
 5    19      0       763  d7d5 d2d4 b8c6 h2h3 e7e6 
 6    12      0      1298  d7d5 d2d4 b8c6 h2h3 e7e5 c2c3 
 7     6      0      2359  d7d5 d2d4 g8f6 f2f3 e7e6 c2c3 
 8   -11      0      7356  d7d5 d2d4 g8f6 b1c3 c8f5 f1d3 e7e6 d3f5 
 9     0      1     13860  d7d5 d2d4 e7e6 b1c3 b8c6 c1d2 g8f6 g1f3 c8d7 
10   -14      2     45761  e7e5 d2d4 e5e4 d4d5 g8f6 b1c3 f8d6 c1d2 
11     0      4    103082  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 b8c6 f1d3 f8d6 e1g1 e8g8 
12   -12      5    133079  d7d5 d2d4 g8f6 g1f3 e7e6 f3e5 b8c6 e5c6 b7c6 b1c3 f6e4 c3e4 
13    -2      8    194120  d7d5 d2d4 g8f6 g1f3 e7e6 f1d3 b8c6 d1e2 f8d6 b1c3 e8g8 e1g1 
14    -7     13    326344  e7e5 d2d4 e5e4 d4d5 g8f6 b1c3 f8d6 f1e2 e8g8 g1h3 f8e8 e1g1 c7c6 
15   -10     20    601562  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 f8d6 f1d3 a7a6 e1g1 e8g8 c1d2 c8d7 f3e5 b8c6 
16   -10     26    805368  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 f8d6 f1d3 a7a6 e1g1 c8d7 c1d2 e8g8 f3e5 b8c6 
17    -9     39   1282884  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 c8d7 f1e2 f8d6 f3e5 b8c6 e5d7 d8d7 e1g1 e8g8 c1d2 f8e8 
18    -9     55   1873354  d7d5 d2d4 g8f6 g1f3 e7e6 b1c3 c8d7 f1e2 f8d6 f3e5 b8c6 e5d7 d8d7 e1g1 e8g8 c1d2 f8e8 
19   -12     85   2965790  d7d5 d2d4 g8f6 g1f3 e7e6 f1e2 f8d6 c2c4 d5c4 e2c4 e8g8 b1c3 b8d7 e1g1 d7b6 c4d3 c8d7 c1d2 d7c6 f3e5 
20   -12    130   4593646  d7d5 d2d4 g8f6 g1f3 e7e6 f1e2 f8d6 c2c4 d5c4 e2c4 e8g8 b1c3 b8d7 e1g1 d7b6 c4d3 c8d7 c1d2 d7c6 f3e5 
21   -12    179   6396786  d7d5 d2d4 g8f6 g1f3 e7e6 f1e2 c7c5 b1c3 f8e7 e1g1 e8g8 c1d2 b8c6 f3e5 c8d7 a2a3 c5c4 e5d7 d8d7 c3b5 
22   -12    341  12560946  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 c1d2 b8c6 f1e2 f6e4 e1g1 e8g8 d1b3 a7a6 f1c1 e4d2 b1d2 c6a5 b3c2 a5c6 
23    -8    454  16747642  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 b1c3 e8g8 f1e2 b8c6 e1g1 f6e4 c1d2 a7a6 g1h1 g8h8 d1e1 f7f5 c4d5 e4d2 e1d2 e6d5 
24    -8    603  21205553  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 b1c3 e8g8 f1e2 b8c6 e1g1 f6e4 c1d2 a7a6 g1h1 g8h8 d1e1 f7f5 c4d5 e4d2 e1d2 e6d5 
25    -9    800  27120003  d7d5 d2d4 g8f6 g1f3 e7e6 c2c4 f8e7 c4d5 e6d5 f1d3 e8g8 b1c3 b8a6 a2a3 c7c5 d3c2 c5c4 e1g1 d8b6 c3a4 b6e6 c1d2 f8d8 
move d7d5

So this would be very interesting on the very computer it is playing the tournaments and then same again for OliThink 5.11.5, because I made changes which affects CPU Cache.

I can open OliThink and give it a depth search if you'd like to compare the output with what you get.

Thank you!
Alternatively you could open you GUI with OliThink with enough time and make die "e3" there.
OliThink can also "analyse" mode but this should be startet after you made "e3" because else it's starting to calculate before already and the output will be different.

I can do the same for Dumb if Richard wishes.

I have said before that I like "Dumb" a lot as a engine, because it is genuine.

[OliverBr]

Thank you, Graham. I am already seeing that everything is ok with your setup.
The error lies within OlIThink, weakness in long time controls combined with no endgame evaluation at all.

PS: OliThink has no endgame evaluation because it was the project from the beginning to just use mobility as eval.
Funnily this seems enough for fast games, but not for slower games. The endgame weakness becomes to big then.