Testing depend largely on good Openings

[Chessqueen]

Rubichess saw f5 immediately
[fen]rn1q1rk1/1b3p1p/p3p1pQ/1p1pP1P1/5P2/BN1B4/P1P4P/b4R1K w - - 0 18 [/fen]
FEN: rn1q1rk1/1b3p1p/p3p1pQ/1p1pP1P1/5P2/BN1B4/P1P4P/b4R1K w - - 0 18

RubiChess-20220223_x86-64-avx2:
1/1 00:00 0 0 -1.57 Nb3-a5
2/2 00:00 0 0 -1.57 Nb3-a5
3/3 00:00 0 0 -1.57 Nb3-a5
4/4 00:00 0 0 -1.57 Nb3-a5
5/5 00:00 0 0 -1.57 Nb3-a5
6/6 00:00 0 0 -1.57 Nb3-a5
7/7 00:00 0 0 -1.57 Nb3-a5
8/8 00:00 0 0 -1.57 Nb3-a5
9/9 00:00 0 0 -1.57 Nb3-a5
10/10 00:00 0 0 -1.57 Nb3-a5
11/11 00:00 0 0 -1.57 Nb3-a5
12/12 00:00 0 0 -1.57 Nb3-a5
13/13 00:00 0 0 -1.57 Nb3-a5
14/14 00:00 0 0 -1.57 Nb3-a5
15/15 00:00 0 0 -1.57 Nb3-a5
16/16 00:00 0 0 -1.57 Nb3-a5
17/17 00:00 0 0 -1.57 Nb3-a5
18/18 00:00 0 0 -1.57 Nb3-a5
19/19 00:00 0 0 -1.57 Nb3-a5
20/20 00:00 0 0 -1.57 Nb3-a5
21/21 00:00 0 0 -1.57 Nb3-a5
22/22 00:00 0 0 -1.57 Nb3-a5
23/23 00:00 0 0 -1.57 Nb3-a5
24/24 00:00 0 0 -1.57 Nb3-a5
25/25 00:00 0 0 -1.57 Nb3-a5
26/26 00:00 0 0 -1.57 Nb3-a5
27/36 00:00 687k 5,407k +4.46 f4-f5 e6xf5 Rf1-f4 Ba1xe5 Rf4-h4 d5-d4+ Kh1-g1 Qd8xg5+ Qh6xg5 f7-f6 Qg5-h6 Rf8-f7 Nb3xd4 Bb7-e4 Nd4-e6 Be4-d5 Ne6-f4 Be5-d4+ Kg1-f1 Bd4-e3 Rh4-h3 Be3xf4 Qh6xf4 Nb8-d7 Ba3-b2 Bd5xa2 Qf4-d6 Nd7-e5 Bb2xe5 f6xe5 Qd6xe5 Ra8-c8 Qe5-a1 Ba2-c4 Qa1xa6
28/47 00:00 1,773k 5,851k +4.52 f4-f5 e6xf5 Rf1-f4 Ba1xe5 Rf4-h4 d5-d4+ Kh1-g1 Qd8xg5+ Qh6xg5 f7-f6 Qg5-h6 Rf8-f7 Nb3xd4 Bb7-e4 Nd4-e6 Be4-d5 Ne6-f4 Be5xf4 Qh6xf4 Nb8-d7 Ba3-b2 Bd5xa2 Qf4-d6 Kg8-g7 Kg1-f1 Kg7-g8 Rh4-h3 Nd7-e5 Bb2xe5 f6xe5 Qd6xe5 Ra8-c8 Qe5-a1 Ba2-c4 Qa1xa6 Bc4xd3+ c2xd3 Rc8-c1+ Kf1-f2 Rc1-c2+ Kf2-f3 Rf7-e7 Qa6-a8+ Kg8-g7

[Lazy_Frank]

This is precisely why so many are switching to unbalanced openings (with color reversal). If you look at the current CCRL Complete Rapid list, the top five engines (four stockfish versions and latest Komodo Dragon) are all within four elo points (!), while the next six (five Stockfish versions or derivatives plus previous Komodo Dragon) are within another dozen Elo of the top five, using reasonably balanced openings. The best engines just don't lose games against each other anymore with good openings (with 4 cpu and Rapid tc). With unbalanced openings, the average result of a two game match between equal opponents is likely to be one win and one draw, or at least close to that. If one engine is just slightly stronger, it will tend to be the one that scores the 1.5. So instead of a four elo gap, we might see a forty elo gap or more when unbalanced openings are used. With good openings, I think it is unlikely that any engine will ever reach 3600 (top now is 3540) on that CCRL Rapid list; at about 2 elo per year, it would take 30 years, but progress will surely get less each year. With unbalanced openings, it is true that there is a practical upper limit of 192 elo on the result between two super engines, assuming that they are strong enough never to lose from the nearly winning side. But as long as engine pairings are limited to maybe 150 elo apart or less, there is no obvious limit to how high ratings might go with unbalanced openings, 5000, 6000, who knows? You may argue that this is artificial, that only "balanced" (i.e. typical GM) openings should be used or else it's not real chess, and that is a valid argument, but unless we go with unbalanced openings computer chess is dying; there isn't enough room for further gains from the normal starting position, the path to a draw is too clear now. We just have to learn to think in terms of pairs of games; as long as each side gets White once from the same opening, it is fair, it's just not how chess is played between humans (well, so far anyway).

I agree on using unbalanced openings to highlight small differences between top engines, but there is no real evidence that the current best engines are 'unbeatable' with balanced openings, here are a couple of examples I noticed in a game analyzed with Dragon 2.6:

Here dragon prefers ..Qd8 (??) without seeing f5! as a continuation, at least in the first minute of analysis
[fen]rnb2rk1/1pqp1p1p/p3p1pQ/4P3/5P2/BN1B4/P1P3PP/b4RK1 b - - 1 2[/fen]


a few moves later, once again the winning f5!! it is not seen by Dragon 2.6
[fen]rn1q1rk1/1b3p1p/p3p1pQ/1p1pP1P1/5P2/BN1B4/P1P4P/b4R1K w - - 2 18[/fen]


What I want to show is that there could be many winning moves that are not seen by the current top engines

Hard to say presented FEN is a opening, for me looks more like a position.
Missing winning moves in some positions that arises from unbalanced openings i would say is a bit more than missing winning moves in some positions from balanced openings.

[lkaufman]

What I want to show is that there could be many winning moves that are not seen by the current top engines

There is no doubt that top engines miss many winning (or drawing) moves; the question is whether they will ever reach positions that allow winning moves for the other side if they can choose their own openings or at least limit their choices to the openings chosen in recent Candidate's or World Championship human tournaments and matches. If playing the Berlin and Nimzo (for example) is always good enough to draw, it means that we must force the engines to play inferior defenses to get decisive results; then it is just a question of how inferior (unbalanced) they have to be. Showing positions where they fail to find the right move is just an argument for unbalanced openings unless the position has occurred in a slow game between top engines on good hardware where they started with either their own or WC human openings.

[Marcus91]

What I want to show is that there could be many winning moves that are not seen by the current top engines

There is no doubt that top engines miss many winning (or drawing) moves; the question is whether they will ever reach positions that allow winning moves for the other side if they can choose their own openings or at least limit their choices to the openings chosen in recent Candidate's or World Championship human tournaments and matches. If playing the Berlin and Nimzo (for example) is always good enough to draw, it means that we must force the engines to play inferior defenses to get decisive results; then it is just a question of how inferior (unbalanced) they have to be. Showing positions where they fail to find the right move is just an argument for unbalanced openings unless the position has occurred in a slow game between top engines on good hardware where they started with either their own or WC human openings.

So you say that this position cannot be obtained without an unbalanced opening? Well, you will be surprised to know that in this game is played by Dragon 2.6 as black, 1 thread, 10 seconds per move. Check it out for yourself, the engine is okay with every move played as black. No opening book is used

[pgn][Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "???"]
[Black "Dragon-2.6-64bit-avx2"]
[Result "*"]
[BlackElo ""]
[Time ""]
[WhiteElo ""]
[TimeControl ""]
[Termination "unterminated"]
[PlyCount "29"]
[WhiteType "program"]
[BlackType "program"]

1. e4 e6 2. Nc3 c5 3. Nge2 a6 4. d4 cxd4 5. Nxd4 Qc7 6. Bd3 Nf6 7. f4 Bb4
8. Nb3 O-O 9. e5 Nd5 10. O-O Nxc3 11. bxc3 Bxc3 12. Ba3 Bxa1 13. Qh5 g6 14.
Qh6 Qd8 15. f5 *

[/pgn]

[lkaufman]

What I want to show is that there could be many winning moves that are not seen by the current top engines

There is no doubt that top engines miss many winning (or drawing) moves; the question is whether they will ever reach positions that allow winning moves for the other side if they can choose their own openings or at least limit their choices to the openings chosen in recent Candidate's or World Championship human tournaments and matches. If playing the Berlin and Nimzo (for example) is always good enough to draw, it means that we must force the engines to play inferior defenses to get decisive results; then it is just a question of how inferior (unbalanced) they have to be. Showing positions where they fail to find the right move is just an argument for unbalanced openings unless the position has occurred in a slow game between top engines on good hardware where they started with either their own or WC human openings.

So you say that this position cannot be obtained without an unbalanced opening? Well, you will be surprised to know that in this game is played by Dragon 2.6 as black, 1 thread, 10 seconds per move. Check it out for yourself, the engine is okay with every move played as black. No opening book is used

[pgn][Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "???"]
[Black "Dragon-2.6-64bit-avx2"]
[Result "*"]
[BlackElo ""]
[Time ""]
[WhiteElo ""]
[TimeControl ""]
[Termination "unterminated"]
[PlyCount "29"]
[WhiteType "program"]
[BlackType "program"]

1. e4 e6 2. Nc3 c5 3. Nge2 a6 4. d4 cxd4 5. Nxd4 Qc7 6. Bd3 Nf6 7. f4 Bb4
8. Nb3 O-O 9. e5 Nd5 10. O-O Nxc3 11. bxc3 Bxc3 12. Ba3 Bxa1 13. Qh5 g6 14.
Qh6 Qd8 15. f5 *

[/pgn]

That is a bit surprising, but all my posts on this topic refer to Rapid (15' + 10" or 40 moves in 15 minutes for CCRL) with at least four threads. So roughly 8 times as many nodes per move as the game you mention. We know that in blitz or even single thread Rapid in CCRL or CEGT top engines do lose games to each other with normal openings, but with four threads and CCRL Rapid tc it almost never happens, and based on the trend I expect that with 16 or more threads and a somewhat slower tc it would be so rare as to never be observed to happen (with either free choice of openings or use of the openings used regularly now at the highest level of human chess).

[Marcus91]

What I want to show is that there could be many winning moves that are not seen by the current top engines

There is no doubt that top engines miss many winning (or drawing) moves; the question is whether they will ever reach positions that allow winning moves for the other side if they can choose their own openings or at least limit their choices to the openings chosen in recent Candidate's or World Championship human tournaments and matches. If playing the Berlin and Nimzo (for example) is always good enough to draw, it means that we must force the engines to play inferior defenses to get decisive results; then it is just a question of how inferior (unbalanced) they have to be. Showing positions where they fail to find the right move is just an argument for unbalanced openings unless the position has occurred in a slow game between top engines on good hardware where they started with either their own or WC human openings.

So you say that this position cannot be obtained without an unbalanced opening? Well, you will be surprised to know that in this game is played by Dragon 2.6 as black, 1 thread, 10 seconds per move. Check it out for yourself, the engine is okay with every move played as black. No opening book is used

[pgn][Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "???"]
[Black "Dragon-2.6-64bit-avx2"]
[Result "*"]
[BlackElo ""]
[Time ""]
[WhiteElo ""]
[TimeControl ""]
[Termination "unterminated"]
[PlyCount "29"]
[WhiteType "program"]
[BlackType "program"]

1. e4 e6 2. Nc3 c5 3. Nge2 a6 4. d4 cxd4 5. Nxd4 Qc7 6. Bd3 Nf6 7. f4 Bb4
8. Nb3 O-O 9. e5 Nd5 10. O-O Nxc3 11. bxc3 Bxc3 12. Ba3 Bxa1 13. Qh5 g6 14.
Qh6 Qd8 15. f5 *

[/pgn]

That is a bit surprising, but all my posts on this topic refer to Rapid (15' + 10" or 40 moves in 15 minutes for CCRL) with at least four threads. So roughly 8 times as many nodes per move as the game you mention. We know that in blitz or even single thread Rapid in CCRL or CEGT top engines do lose games to each other with normal openings, but with four threads and CCRL Rapid tc it almost never happens, and based on the trend I expect that with 16 or more threads and a somewhat slower tc it would be so rare as to never be observed to happen (with either free choice of openings or use of the openings used regularly now at the highest level of human chess).

What about FGRL? Using 8moves_v3 book sf15 win against Dragon 3 (+13 =137 -0) in 60' + 15" 1 Thread, again we have no evidence on the elo limit on balanced openings. However using unbalanced openings It is totally legitimate and helps us to understand which of two strong engines is the best without falling into many draw.
I admit that it is a bit strange to see +0 =68 -0 in ccrl 40/15, but +13 =137 -0 in 60'+15"

[lkaufman]

What I want to show is that there could be many winning moves that are not seen by the current top engines

There is no doubt that top engines miss many winning (or drawing) moves; the question is whether they will ever reach positions that allow winning moves for the other side if they can choose their own openings or at least limit their choices to the openings chosen in recent Candidate's or World Championship human tournaments and matches. If playing the Berlin and Nimzo (for example) is always good enough to draw, it means that we must force the engines to play inferior defenses to get decisive results; then it is just a question of how inferior (unbalanced) they have to be. Showing positions where they fail to find the right move is just an argument for unbalanced openings unless the position has occurred in a slow game between top engines on good hardware where they started with either their own or WC human openings.

So you say that this position cannot be obtained without an unbalanced opening? Well, you will be surprised to know that in this game is played by Dragon 2.6 as black, 1 thread, 10 seconds per move. Check it out for yourself, the engine is okay with every move played as black. No opening book is used

[pgn][Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "?"]
[White "???"]
[Black "Dragon-2.6-64bit-avx2"]
[Result "*"]
[BlackElo ""]
[Time ""]
[WhiteElo ""]
[TimeControl ""]
[Termination "unterminated"]
[PlyCount "29"]
[WhiteType "program"]
[BlackType "program"]

1. e4 e6 2. Nc3 c5 3. Nge2 a6 4. d4 cxd4 5. Nxd4 Qc7 6. Bd3 Nf6 7. f4 Bb4
8. Nb3 O-O 9. e5 Nd5 10. O-O Nxc3 11. bxc3 Bxc3 12. Ba3 Bxa1 13. Qh5 g6 14.
Qh6 Qd8 15. f5 *

[/pgn]

That is a bit surprising, but all my posts on this topic refer to Rapid (15' + 10" or 40 moves in 15 minutes for CCRL) with at least four threads. So roughly 8 times as many nodes per move as the game you mention. We know that in blitz or even single thread Rapid in CCRL or CEGT top engines do lose games to each other with normal openings, but with four threads and CCRL Rapid tc it almost never happens, and based on the trend I expect that with 16 or more threads and a somewhat slower tc it would be so rare as to never be observed to happen (with either free choice of openings or use of the openings used regularly now at the highest level of human chess).

What about FGRL? Using 8moves_v3 book sf15 win against Dragon 3 (+13 =137 -0) in 60' + 15" 1 Thread, again we have no evidence on the elo limit on balanced openings. However using unbalanced openings It is totally legitimate and helps us to understand which of two strong engines is the best without falling into many draw.
I admit that it is a bit strange to see +0 =68 -0 in ccrl 40/15, but +13 =137 -0 in 60'+15"

This discrepancy is a bit puzzling. I looked at a sample of the openings in that 8moves_v3 book, and while most looked reasonably normal (White edge not too far from his initial edge), a few were pretty good for White, SF evals close to a pawn plus (roughly double White's initial edge). It's a fairly old set, probably all the openings evaluated as reasonable at the time, but now standards are very different thanks to NNUE. So my best guess is that just enough of the openings were sufficiently unbalanced to result in 9% wins. Perhaps CCRL used a book where the openings were more typical of human GM play or at least closer to equal? This shows a real problem; the choice of opening book now has a major influence on ratings; the more unbalanced the book, the more it favors the stronger engine (assuming the engines are very strong and not too far apart in strength). Books are not either "balanced" or "unbalanced", they come in all shades in between as well. There is even a huge difference between books that aim for nearly equal positions and books that aim for a par White advantage (which is more appropriate in my opinion).

[Modern Times]

I admit that it is a bit strange to see +0 =68 -0 in ccrl 40/15, but +13 =137 -0 in 60'+15"

You would have to confirm with Graham, but I think 52 of those 68 games came from his IECGMasters.cgb book, which he notes as
"Made from games between players rated 2400+ in the IECG". It appears to be 8 moves deep.

[lkaufman]

I admit that it is a bit strange to see +0 =68 -0 in ccrl 40/15, but +13 =137 -0 in 60'+15"

You would have to confirm with Graham, but I think 52 of those 68 games came from his IECGMasters.cgb book, which he notes as
"Made from games between players rated 2400+ in the IECG". It appears to be 8 moves deep.

Games played by strong correspondence players would presumably feature very correct openings, so this would be what I would prefer to call "normal" or "standard" openings, though most people call them "balanced". That is probably the explanation of the discrepancy, the 8 move book used by FGRL is much more diverse, some rather balanced openings, some that no one would play in top level correspondence as Black.

[Uri Blass]

The main question if there are equal positions that give both sides better chances to win so the better engine can win.
Note that not all the drawn equal positions are the same

You may try some unbalanced positions when one side has more material but the opponent has a positional compensation and maybe white and black will get 50% but there will be less draws if you play stockfish vs stockfish with 1:10 time handicap.