Stockfish Natural TB loses heavily to Stockfish master

[Laskos]

After seeing that NTB3 plays perfectly on root TB positions, I proceeded with openings out of TBs. I took 6-men White Wins as openings, but only 5-men Syzygy.

1000 games from 6-men Wins with 5-men TBs at 0.25s per move (no time losses):
Score of SF_Early_Mate2 vs SF_NTB3: 461 - 436 - 103 [0.512] 1000
ELO difference: 8.69 +/- 20.39
Finished match

Here is the first problem: correct pentanomial error margins are 4.88 ELO points 1.95 standard deviations, so the result is:
ELO difference: 8.69 +/- 4.88

It seems NTB3 slightly loses ELO points to early_mate2 in close to TB openings (but out of TB). Maybe something with TB probing during the search.
===================================


Won games length:

Stockfish early_mate2 Wins:
Mean game length: 36.7 moves
Median game length: 34 moves

Stockfish NTB3 Wins:
Mean game length: 41.4 moves
Median game length: 38 moves

Significantly shorter paths to Win for early_mate2 compared to NTB3.
===========================================


Total Mates resolved from the start to the end of the games:
Stockfish early_mate2: 11095
Stockfish NTB3: 10406

Mates resolved in the first 10 moves of the games:
Stockfish early_mate2: 480
Stockfish NTB3: 412

Slightly more mates resolved by early_mate2 compared to NTB3.
===========================================

All in all, better results with early_mate, and that ELO loss of NTB3, if not a statistical fluke or compile issue, must be addressed. Will check a bit the PGN for missed Wins of NTB3, but it will probably be hard to see something conclusive, because the openings are out of TBs, and when the root position arrives in TBs, NTB3 plays perfectly.

[mcostalba]

Thanks Kai for the test.

Test was aimed at position starting from TB and it was ok it seems.

Regarding other tests, starting from a position just before TB, sorry but this does not make a lot of sense to me. If you want to compare for instance on mate length then please test from starting position on normal games, because I think NTB will find the mate much earlier then early mate in normal games (considering how normal search is implemented in NTB compared to current master and early_mate).

[Laskos]

Thanks Kai for the test.

Test was aimed at position starting from TB and it was ok it seems.

Regarding other tests, starting from a position just before TB, sorry but this does not make a lot of sense to me. If you want to compare for instance on mate length then please test from starting position on normal games, because I think NTB will find the mate much earlier then early mate in normal games (considering how normal search is implemented in NTB compared to current master and early_mate).

Yes, seems like a good idea. If from highly sensitive to TB implementation 6-men Wins suite with 5-men TBs, I found 8 ELO points loss for NTB3, then guess how much this loss in ELO would be from 2moves_v1.pgn? Probably some 0.3 ELO points. How do you want to measure that?
Position is not "just before TB", to enter the TB the engines need usually from 15 to 30 moves. That's why I posted the number of mates resolved after only 10 moves were played, when almost all the mates found were on positions outside TBs. For only 3 moves played, where literally all positions were 6-men (outside 5-men TBs), the situation seems even more skewed, with 22 mates resolved by early_mate and 12 by NTB3. From 2moves_v1, only a small fraction of games relies on TBs in general, and a much smaller fraction on TB implementation. If you didn't change the master to such a degree that it now plays differently even without any TBs, we will need maybe 10 or 100 or 1000 times more games from 2moves_v1 panacea suite to have a statistical significance comparable to what I achieve with my opening suite. Do you realize that it is acting like a magnifying glass for what you are trying to achieve with NTB?

[Laskos]

Thanks Kai for the test.

Test was aimed at position starting from TB and it was ok it seems.

Regarding other tests, starting from a position just before TB, sorry but this does not make a lot of sense to me. If you want to compare for instance on mate length then please test from starting position on normal games, because I think NTB will find the mate much earlier then early mate in normal games (considering how normal search is implemented in NTB compared to current master and early_mate).

Yes, seems like a good idea. If from highly sensitive to TB implementation 6-men Wins suite with 5-men TBs, I found 8 ELO points loss for NTB3, then guess how much this loss in ELO would be from 2moves_v1.pgn? Probably some 0.3 ELO points. How do you want to measure that?
Position is not "just before TB", to enter the TB the engines need usually from 15 to 30 moves. That's why I posted the number of mates resolved after only 10 moves were played, when almost all the mates found were on positions outside TBs. For only 3 moves played, where literally all positions were 6-men (outside 5-men TBs), the situation seems even more skewed, with 22 mates resolved by early_mate and 12 by NTB3. From 2moves_v1, only a small fraction of games relies on TBs in general, and a much smaller fraction on TB implementation. If you didn't change the master to such a degree that it now plays differently even without any TBs, we will need maybe 10 or 100 or 1000 times more games from 2moves_v1 panacea suite to have a statistical significance comparable to what I achieve with my opening suite. Do you realize that it is acting like a magnifying glass for what you are trying to achieve with NTB?

As I don't see any wrongdoings with my methodology, I continue with maybe the last important, hard to quantify indicator: "silliness", from which maybe all this "naturalness" came from. What is that "silliness" in computer chess I don't know, but let's build a model.

To quantify "silliness" as seen by a strong human player, I took Fruit 2.1, an average engine, which is GM level, but at the time control per position I used of 0.25s/move, behaves roughly as a pretty strong human. Then, in 1000 games at 0.25s/move, I got the number of moves of SF early_mate and SF NTB3 games the engine Fruit considers blunders by a margin 3-12 pawns for total number of "silly" moves, and 9-12 pawns for very "silly", usually Queen "blunders". Sure, almost all of them are not really "blunders", SF + Syzygy-5 on 6-men knows much better than Fruit or a strong human what it is doing, but let's consider these para-blunders as "silly" (this is a model).

Early_mate2:
1000 games

Silly moves in the range of N,B,R,Q "blunders" according to Fruit:
762

Very silly moves (Queen "blunders") according to Fruit:
46





NTB3:
1000 games

Silly moves in the range of N,B,R,Q "blunders" according to Fruit:
883

Very silly moves (Queen "blunders") according to Fruit:
100

=====================================



Unfortunately, at this indicator too, NTB3 fares worse than early_mate2. Maybe I have to count also "silliness" of TB implementation in the first 5 moves after exiting 2moves_v1.pgn (a masterpiece of openings, by the way), as Marco suggests?

[Laskos]

Finally, I got failures of NTB3 in 6-men TB positions at the root with a large 6-men reasonably hard suite of Wins (1945 positions, 3890 games side and reversed). They are very rare, but they do happen:

Score of SF_Early_Mate2 vs SF_NTB3: 1945 - 1942 - 3 [0.500] 3890
ELO difference: 0.27 +/- 10.90
Finished match

3 failures to convert on root 6-men positions out of 1945 6-men Wins with 6-men Syzygy. Here are they:

[pgn][Event "?"]
[Site "?"]
[Date "2017.09.20"]
[Round "1454"]
[White "SF_NTB3"]
[Black "SF_Early_Mate2"]
[Result "1/2-1/2"]
[FEN "8/1P5p/1KQ5/8/7k/8/1q6/8 w - - 0 1"]
[PlyCount "34"]
[SetUp "1"]
[TimeControl "0.1/move"]

1. Qb5 {+102.00/10 0.10s} Qf6+ {-132.79/11 0.099s} 2. Ka5 {+103.00/11 0.10s}
Qd8+ {-132.79/13 0.10s} 3. Ka4 {+104.00/12 0.10s} Qd1+ {-132.79/14 0.10s}
4. Qb3 {+105.00/11 0.11s} Qa1+ {-132.79/13 0.10s} 5. Kb5 {+106.00/12 0.099s}
Qe5+ {-132.79/13 0.10s} 6. Kb6 {+107.00/13 0.10s} Qd6+ {-132.79/14 0.10s}
7. Ka5 {+108.00/15 0.10s} Qc5+ {-132.79/14 0.10s} 8. Qb5 {+110.00/14 0.10s}
Qa7+ {-132.79/15 0.10s} 9. Kb4 {+114.00/13 0.10s} Qb8 {-132.79/16 0.11s}
10. Qc6 {+115.00/12 0.10s} Kg3 {-132.79/13 0.10s} 11. Kb3 {+116.00/11 0.10s}
Kh2 {-132.79/16 0.10s} 12. Qh6+ {+118.00/14 0.099s} Kg2 {-132.79/16 0.099s}
13. Qg7+ {+119.00/15 0.10s} Kf1 {-132.79/18 0.10s} 14. Qf6+ {+120.00/19 0.10s}
Kg1 {-132.79/20 0.10s} 15. Qg7+ {+119.00/15 0.10s} Kf1 {-132.79/21 0.10s}
16. Qf6+ {+120.00/12 0.10s} Kg1 {-132.79/18 0.10s} 17. Qg7+ {+119.00/15 0.10s}
Kf1 {-132.79/47 0.10s, Draw by 3-fold repetition} 1/2-1/2[/pgn]


[pgn][Event "?"]
[Site "?"]
[Date "2017.09.20"]
[Round "1532"]
[White "SF_NTB3"]
[Black "SF_Early_Mate2"]
[Result "1/2-1/2"]
[FEN "8/1b3k2/1r6/3pK3/6Q1/8/8/8 w - - 0 1"]
[PlyCount "20"]
[SetUp "1"]
[TimeControl "0.1/move"]

1. Qd7+ {+110.00/9 0.14s} Kg6 {-132.79/13 0.10s} 2. Qd8 {+111.00/11 0.10s}
Rb2 {-132.79/12 0.10s} 3. Qg8+ {+112.00/11 0.10s} Kh6 {-132.79/14 0.10s}
4. Qg3 {+113.00/12 0.10s} Kh5 {-132.79/13 0.10s} 5. Kd6 {+114.00/10 0.13s}
d4 {-132.79/13 0.11s} 6. Qe5+ {+119.00/9 0.10s} Kg6 {-132.79/13 0.10s}
7. Qg3+ {+120.00/14 0.11s} Kf5 {-132.79/13 0.10s} 8. Qe5+ {+119.00/12 0.10s}
Kg6 {-132.79/14 0.10s} 9. Qe6+ {+120.00/13 0.10s} Kg5 {-132.79/14 0.10s}
10. Qe5+ {+119.00/21 0.10s} Kg6 {-132.79/42 0.10s, Draw by 3-fold repetition}
1/2-1/2[/pgn]



[pgn][Event "?"]
[Site "?"]
[Date "2017.09.20"]
[Round "2028"]
[White "SF_NTB3"]
[Black "SF_Early_Mate2"]
[Result "1/2-1/2"]
[FEN "8/4Q3/8/3r1k2/5p2/2Kp4/8/8 w - - 0 1"]
[PlyCount "11"]
[SetUp "1"]
[TimeControl "0.1/move"]

1. Qe1 {+102.00/10 0.10s} d2 {0.00/25 0.10s} 2. Qd1 {0.00/36 0.10s} Ke4 {0.00/26 0.10s}
3. Qe2+ {0.00/25 0.11s} Kf5 {0.00/30 0.099s} 4. Qd1 {0.00/38 0.099s}
Ke4 {0.00/28 0.10s} 5. Qe2+ {0.00/34 0.10s} Kf5 {0.00/32 0.10s}
6. Qd1 {0.00/40 0.10s, Draw by 3-fold repetition} 1/2-1/2[/pgn]


Also, I got an even more conclusive ELO loss of NTB3 to early_mate 2 from another suite of 6-men Wins using 5-men Syzygy:
0.25s/move

Score of SF_Early_Mate2 vs SF_NTB3: 1637 - 1543 - 1074 [0.511] 4254
ELO difference: 7.68 +/- 9.02
Finished match

ELO difference: 7.68 +/- 2.89 using correct pentanomial variance.

That's hardly a fluke. Even if compile speeds differ by say at most 5%, the suite has another nice property that it compresses this sort of differences by a factor of 4-6 compared to regular openings. So, if 5% speed is equivalent to 5-10 ELO points at this time control from regular openings, with this suite of 6-men Wins, it would give only 1-2 ELO points difference. I strongly believe that NTB3 is ELO-losing implementation, and as we saw above, does not convert all TB Wins at the root.

[Michel]

The sad thing is that master already deals with the 3-fold reps issue with an elegant and provably correct method (the proof is 3 lines) and which still permits the engine to steer the search (to have natural play).

I wonder if Marco will ever come up with something that is of equal simplicity and provably correct at the same time.

[syzygy]

Interestingly, these games expose a cosmetic bug in SF-master: if the next move achieves a 3-fold repetition, it should be displayed with a draw score rather than the TB loss score.

It seems this can be fixed by calling pos.is_draw() to check for a draw by repetition before calling probe_dtz() on a root move.

[Laskos]

The sad thing is that master already deals with the 3-fold reps issue with an elegant and provably correct method (the proof is 3 lines) and which still permits the engine to steer the search (to have natural play).

I wonder if Marco will ever come up with something that is of equal simplicity and provably correct at the same time.

I left overnight play many games at 0.25s/move, because the failures of NTB3 with TBs at root are rare, but they do occur.
Moderately hard 6-men suite of White Wins (4549 unique positions) as opening positions with 6-men Syzygy on SSD.

9098 games (2 x 4549 side and reversed)

Score of SF_Early_Mate2 vs SF_NTB3: 4563 - 4495 - 40 [0.504] 9098
ELO difference: 2.60 +/- 7.11
Finished match

SF_early_mate won all of its 4549 6-men TB wins.
SF_NTB3 not only drew 40 out of its 4549 6-men wins, it managed to lose 14 6-men wins.

Looking at the PGN, there are two sorts of failures of NTB3 I saw:
As before, 3-fold repetitions due to what you discuss, probalby not playing correctly DTZ optimal moves.
Posting 2 such draws:


[pgn]
[Event "?"]
[Site "?"]
[Date "2017.09.21"]
[Round "1234"]
[White "SF_NTB3"]
[Black "SF_Early_Mate2"]
[Result "1/2-1/2"]
[PlyCount "24"]
[SetUp "1"]
[TimeControl "0.25/move"]
[FEN "1K6/1P4p1/6k1/2q5/8/8/7Q/8 w - - 0 1"]

1.Qg3+ {+99.00/14 0.25s} 1...Kf6 {-132.79/14 0.25s} 2.Qb3 {+102.00/14
0.25s} 2...Ke7 {-132.79/17 0.25s} 3.Qb2 {+107.00/16 0.25s} 3...g5 {
-132.79/18 0.25s} 4.Qe2+ {+104.00/15 0.25s} 4...Kf8 {-132.79/20 0.25s} 5.
Qa2 {+108.00/21 0.25s} 5...Qe5+ {-132.79/21 0.25s} 6.Ka8 {+111.00/20 0.25s
} 6...Qe4 {-132.79/21 0.25s} 7.Qa3+ {+112.00/21 0.25s} 7...Ke8 {-132.79/22
0.25s} 8.Qc5 {+115.00/21 0.25s} 8...Kd7 {-132.79/26 0.25s} 9.Qc8+ {
+120.00/20 0.25s} 9...Ke7 {-132.79/27 0.25s} 10.Qc5+ {+115.00/22 0.25s}
10...Kd7 {-132.79/27 0.25s} 11.Qc8+ {+120.00/25 0.25s} 11...Ke7 {
-132.79/28 0.25s} 12.Qc5+ {+115.00/21 0.25s} 12...Kd7 {-132.79/62 0.25s,
Draw by 3-fold repetition} 1/2-1/2
[/pgn]


[pgn]
[Event "?"]
[Site "?"]
[Date "2017.09.21"]
[Round "3452"]
[White "SF_NTB3"]
[Black "SF_Early_Mate2"]
[Result "1/2-1/2"]
[PlyCount "48"]
[SetUp "1"]
[TimeControl "0.25/move"]
[FEN "8/1P5p/1K5Q/8/8/1q4k1/8/8 w - - 0 1"]

1.Ka7 {+91.00/12 0.25s} 1...Qa3+ {-132.79/13 0.25s} 2.Qa6 {+92.00/15 0.25s
} 2...Qc5+ {-132.79/15 0.25s} 3.Ka8 {+93.00/14 0.25s} 3...Qd5 {-132.79/17
0.25s} 4.Qa3+ {+94.00/17 0.25s} 4...Kh2 {-132.79/16 0.25s} 5.Ka7 {
+95.00/17 0.25s} 5...Qd4+ {-132.79/16 0.25s} 6.Ka6 {+96.00/15 0.25s} 6...
Qf6+ {-132.79/18 0.25s} 7.Kb5 {+97.00/17 0.25s} 7...Qe5+ {-132.79/18 0.25s
} 8.Kc6 {+98.00/18 0.25s} 8...Qe8+ {-132.79/22 0.25s} 9.Kb6 {+99.00/20
0.25s} 9...Qd8+ {-132.79/21 0.25s} 10.Kb5 {+100.00/18 0.25s} 10...Qd5+ {
-132.79/21 0.25s} 11.Ka6 {+102.00/20 0.25s} 11...Qe6+ {-132.79/21 0.25s}
12.Ka7 {+105.00/20 0.25s} 12...Qf7 {-132.79/24 0.25s} 13.Qb2+ {+106.00/18
0.25s} 13...Kg3 {-132.79/23 0.25s} 14.Kb6 {+110.00/16 0.25s} 14...Qe6+ {
-132.79/16 0.25s} 15.Ka5 {+111.00/18 0.25s} 15...Qe1+ {-132.79/17 0.25s}
16.Ka4 {+114.00/20 0.25s} 16...Qe8+ {-132.79/19 0.25s} 17.Qb5 {+115.00/18
0.25s} 17...Qb8 {-132.79/18 0.25s} 18.Qb3+ {+116.00/18 0.25s} 18...Kf2 {
-132.79/20 0.25s} 19.Qf7+ {+118.00/15 0.25s} 19...Kg1 {-132.79/21 0.25s}
20.Qg7+ {+119.00/17 0.25s} 20...Kf1 {-132.79/23 0.25s} 21.Qf7+ {+120.00/18
0.25s} 21...Kg1 {-132.79/21 0.25s} 22.Qg7+ {+119.00/19 0.25s} 22...Kf1 {
-132.79/20 0.25s} 23.Qf6+ {+120.00/20 0.25s} 23...Kg2 {-132.79/22 0.25s}
24.Qg7+ {+119.00/19 0.25s} 24...Kf1 {-132.79/47 0.25s, Draw by 3-fold
repetition} 1/2-1/2
[/pgn]


But all the losses in won TB positions and a part of the draws were due to something strange occuring with NTB3, but never with master or early_mate:


[pgn]
[Event "?"]
[Site "?"]
[Date "2017.09.21"]
[Round "2908"]
[White "SF_NTB3"]
[Black "SF_Early_Mate2"]
[Result "1/2-1/2"]
[PlyCount "34"]
[SetUp "1"]
[TimeControl "0.25/move"]
[FEN "6Q1/8/5k2/5p2/K2p4/3r4/8/8 w - - 0 1"]

1.Qg1 {0.40s} 1...Re3 {0.00/12 0.26s} 2.Qg2 {0.00/24 0.30s} 2...Ke5 {
0.00/16 0.25s} 3.Kb5 {0.00/61 0.26s} 3...f4 {0.00/20 0.25s} 4.Qg4 {0.00/76
0.25s} 4...f3 {0.00/15 0.25s} 5.Qh4 {0.00/71 0.25s} 5...Rb3+ {0.00/18
0.25s} 6.Kc4 {0.00/127 0.001s} 6...Rc3+ {0.00/20 0.25s} 7.Kb4 {0.00/127
0.001s} 7...Re3 {0.00/23 0.25s} 8.Qh2+ {0.00/107 0.25s} 8...Ke4 {0.00/26
0.25s} 9.Qd2 {0.00/127 0.12s} 9...Re2 {0.00/23 0.25s} 10.Qc1 {0.00/67
0.25s} 10...f2 {0.00/25 0.25s} 11.Kc4 {0.00/21 0.26s} 11...Re1 {0.00/28
0.25s} 12.Qc2+ {0.00/41 0.25s} 12...Ke3 {0.00/31 0.25s} 13.Qd3+ {0.00/47
0.25s} 13...Kf4 {0.00/34 0.25s} 14.Qd2+ {0.00/42 0.25s} 14...Kf3 {0.00/35
0.25s} 15.Qd3+ {0.00/47 0.25s} 15...Kf4 {0.00/37 0.25s} 16.Qd2+ {0.00/49
0.25s} 16...Kf3 {0.00/37 0.25s} 17.Qd3+ {0.00/52 0.25s} 17...Kf4 {0.00/39
0.25s, Draw by 3-fold repetition} 1/2-1/2
[/pgn]



[pgn]
[Event "?"]
[Site "?"]
[Date "2017.09.21"]
[Round "2904"]
[White "SF_NTB3"]
[Black "SF_Early_Mate2"]
[Result "0-1"]
[PlyCount "20"]
[SetUp "1"]
[TimeControl "0.25/move"]
[FEN "6Q1/8/4r3/4k1P1/4n1K1/8/8/8 w - - 0 1"]

1.g6 {2.2s} 1...Nf6+ 2.Kf3 {-119.00/22 0.25s} 2...Nxg8 {+M27/27 0.25s} 3.
g7 {-M18/21 0.25s} 3...Kf5 {+M15/29 0.25s} 4.Kg3 {-M14/28 0.25s} 4...Ke4 {
+M13/34 0.25s} 5.Kf2 {-M12/32 0.25s} 5...Kf4 {+M11/38 0.25s} 6.Kg2 {
-M10/40 0.25s} 6...Re2+ {+M9/52 0.25s} 7.Kf1 {-M8/80 0.25s} 7...Kf3 {
+M7/94 0.25s} 8.Kg1 {-M6/127 0.17s} 8...Re1+ {+M5/102 0.25s} 9.Kh2 {
-M4/127 0.008s} 9...Ra1 {+M3/127 0.064s} 10.Kh3 {-M2/127 0.003s} 10...Rh1#
{+M1/127 0.004s, Black mates} 0-1
[/pgn]

Observe the first move of NTB3 in these games (failures in converting TB wins): it somehow doesn't hit the root TB, oversteps its allotted time, which doesn't give a loss on time just because I set timemargin to 10 seconds in Cutechess-cli to not have any time losses. My 6-men Syzygy are on a pretty fast SSD, there seemed to be nothing wrong with my system generally, and it never happened to me before with SF master and its improved variation, early_mate.
I also checked the settings of engines (Hash, overhead, paths, etc), they are identical for NTB3 and early_mate. It seems to me (if my compile of NTB3 is not a bit broken, but I don't believe that) like another problem with NTB3, of different kind.

[Michel]

Not only is Marco abusing the PR(=pull request) system for continuously posting new versions of NTB, each time promising it to be the best one ever (we are now at the 5th version), but he is now also instating censorship

https://github.com/official-stockfish/S ... /pull/1257

Note that Marco made no attempt in the previous PR's to address in a sensible manner any of the technical issues that were raised with his approaches, instead relying on philosophical arguments and (incorrect) hand waving.

I know we are not supposed to discuss politics here but the behaviour of a certain high ranking American official comes to mind.

[syzygy]

Reminds me more of a former Iraqi government speaker.

His version fails before his eyes (at least if he has seen the games Kai has posted here) in exactly the way it was predicted (plus in some more ways), yet he continues to deny that this is possible.