Ronald: Yes, the result is inconclusive, and the change, as I understand, was cosmetical. It does bring some decrease in depth of 0.2 plies, probably statistically significant (2 standard deviations would be something like 3 or 4 over sqrt(2000) ~ 0.1 ply), but I don't know if depth here means something.
Ferdy: after waking up at 5 in the morning, I too wanted too see the scaling with time control, before I saw your post. Also, I started to suspect that NTB2 will fail to solve even some hard 5-men positions, if it fails with 6-men. Yes, it fails to solve at 15''+ 0.15'' time control most of the hard 5-men wins against Master Syzygy. I now used Cutechess-Cli, because I already collected in LittleBlitzer all relevant stats like time used, nps, depth and the time losses.
100 games
Suite: Hard 5-men White wins
TC: 15''+ 0.15'':
Score of SF Master vs SF NTB2: 50 - 4 - 46 [0.730] 100
ELO difference: 172.78 +/- 49.99
Finished match
Here is the PGN of these 100 games
http://s000.tinyupload.com/?file_id=913 ... 3466545204
Please check the PGN, the performance of SF NTB2 is so atrocious, that I even started to suspect I have something broken (either compiles, or TBs). I remember Scorpio WDL egbb's in Houdini some 5 years ago performing better than that.
SF NTB2 failed to solve 46 out of 50 of hard 5-men wins.
Here are these 46 positions:
Code: Select all
1B4B1/1n6/8/8/k7/6K1/8/8 w - -
1B6/8/5K2/3B4/8/8/2k5/5n2 w - -
1K4B1/8/6n1/8/7B/8/1k6/8 w - -
2K5/3B4/8/6B1/8/8/2n5/1k6 w - -
2K5/8/3B4/k7/6n1/8/2B5/8 w - -
3BK3/8/8/5B2/8/5k2/8/n7 w - -
3N4/8/7k/8/2K5/1B6/8/n7 w - -
3k2B1/3n4/8/8/4K3/8/8/4B3 w - -
3k4/8/1n6/7K/8/2B5/B7/8 w - -
3k4/8/8/8/2K2n2/B6B/8/8 w - -
4B3/6n1/8/8/1K5k/8/8/6B1 w - -
4k3/8/8/8/8/2B5/2K4n/1B6 w - -
5n2/8/8/8/5K2/1B6/2N5/6k1 w - -
6BB/k7/1n6/6K1/8/8/8/8 w - -
6K1/5B2/8/5b2/8/8/6N1/3k4 w - -
6k1/8/B7/8/8/2K4n/7B/8 w - -
7B/8/8/1n6/8/8/k1B5/3K4 w - -
7k/8/8/n7/8/1B6/4K3/5N2 w - -
7n/6k1/8/8/8/B7/8/1K5B w - -
8/1k6/3K4/8/7P/8/4R3/2r5 w - -
8/1kB5/6K1/8/8/8/4n3/1B6 w - -
8/1n6/2k5/8/6B1/4BK2/8/8 w - -
8/1n6/8/8/7K/k7/8/1B2B3 w - -
8/3KB3/8/8/8/8/n1B5/6k1 w - -
8/3n4/8/8/8/2N5/2B5/4K2k w - -
8/6B1/4k3/1BK5/8/8/7n/8 w - -
8/6B1/8/8/8/N3K2n/8/2k5 w - -
8/7B/8/1n4K1/8/6B1/k7/8 w - -
8/8/1K6/8/1B6/8/2B5/5kn1 w - -
8/8/1k6/8/2B2n2/8/3K4/B7 w - -
8/8/2K5/8/3k1B2/8/8/n2B4 w - -
8/8/3n3K/8/8/5B2/5B2/1k6 w - -
8/8/8/2B1k3/8/8/8/1Bn1K3 w - -
8/8/8/4K3/1B6/8/1N5k/n7 w - -
8/8/8/5B2/5K2/1k6/8/n5B1 w - -
8/8/8/8/3K4/8/3k4/B2B3n w - -
8/B3k3/8/2K5/5b2/2N5/8/8 w - -
8/K5k1/8/5B2/8/n7/8/4N3 w - -
8/kB1n2K1/8/8/8/2B5/8/8 w - -
B7/8/1B1n4/K7/8/8/8/5k2 w - -
K7/3B4/8/7r/8/8/4k3/7N w - -
R7/1r6/7k/8/8/8/2PK4/8 w - -
k7/8/8/2K5/8/4B1N1/8/4n3 w - -
k7/8/8/7R/8/8/3r3P/7K w - -
k7/8/8/8/2N1K3/8/1B6/6n1 w - -
kn6/3B4/8/8/8/B4K2/8/8 w - -
Going to longer time control:
100 games
Suite: Hard 5-men White wins
TC: 60''+ 0.6'':
Score of SF Master vs SF NTB2: 51 - 14 - 35 [0.685] 100
ELO difference: 134.95 +/- 57.15
Finished match
There was one loss on time from SF NTB2.
NTB2 now fails to solve 35 out of 50 positions. Which is an improvement
.
Here are these 35 positions:
Code: Select all
1B3k2/8/8/8/6n1/2K5/8/7B w - -
1B6/3K4/8/8/1k6/1B6/6n1/8 w - -
1K4B1/8/6n1/8/7B/8/1k6/8 w - -
1K6/8/8/2n5/8/8/1k6/3BB3 w - -
1k6/8/4B3/N7/3K4/8/7n/8 w - -
2B2n2/8/5Bk1/8/8/3K4/8/8 w - -
2k5/K6B/8/8/8/8/1n1B4/8 w - -
3B4/4K3/4B3/8/8/8/7k/5n2 w - -
3k2B1/3n4/8/8/4K3/8/8/4B3 w - -
4B3/6n1/8/8/1K5k/8/8/6B1 w - -
4B3/n7/8/5k2/8/4K3/3B4/8 w - -
5B2/5B2/k7/8/8/7K/2n5/8 w - -
5B2/8/6B1/6K1/8/8/4k3/n7 w - -
5B2/8/8/1k6/4B3/1n6/5K2/8 w - -
5K2/7B/1k6/8/8/8/1B4n1/8 w - -
5n2/8/8/7k/8/7B/8/2B3K1 w - -
5n2/8/B1K5/8/8/8/k5N1/8 w - -
6k1/n7/1N6/8/3K4/8/5B2/8 w - -
7B/5k2/7n/3K4/8/3B4/8/8 w - -
7B/8/8/1n6/8/8/k1B5/3K4 w - -
7k/8/8/8/7n/BB1K4/8/8 w - -
8/2B2n2/8/8/8/2K5/8/5Bk1 w - -
8/3B4/2K5/8/8/8/5k2/1nB5 w - -
8/3KB3/8/8/8/8/n1B5/6k1 w - -
8/8/1k6/4r3/8/8/5P2/2R4K w - -
8/8/2B3K1/2B5/8/3n4/k7/8 w - -
8/8/2k5/7B/2nBK3/8/8/8 w - -
8/8/2n1K3/B7/8/7B/8/2k5 w - -
8/8/3n1K2/BB6/8/5k2/8/8 w - -
8/8/8/3B4/8/5K1n/3N3k/8 w - -
8/8/8/6K1/8/8/1BB5/4k2n w - -
8/8/8/7B/K4B2/8/8/5kn1 w - -
8/B7/6k1/5n2/8/7K/8/5B2 w - -
B3K3/4B3/7n/8/8/8/8/6k1 w - -
k3Bn2/8/8/8/K7/4B3/8/8 w - -
And after I read Ferdy's post, I tested at 120''+ 1.2'', the result was
Score of SF Master vs SF NTB2: 50 - 23 - 27 [0.635] 100
ELO difference: 96.19 +/- 60.18
Finished match
SF NTB2 failed to solve now 27 out of 50 positions, which is again an improvement. But at this rate even SF No TB will improve with TC. If I am not doing something wrong, Marco would better abandon this whole "Natural TB" idea. If he doesn't like Master Syzygy (they are probably "Unnatural"), he would better go for no TBs at all.