Does Stockfish really need syzygy rtbz-files at all?

[Jouni]

Inspired by Kai's tests I was curious how SF8 plays with 5 piece syzygy and only WDL = rtbw-files. After 3000 selfplay games I got almost same +6 ELO as with full syzygy set! Previously I think engines with bitbases (wdl) have difficulties to benefit anything. Maybe SF is now different.Can You post just one 5 piece position, where SF8 cannot convert win without rtbz-file?

[syzygy]

You could try playing SF+WDL+DTZ against SF+WDL.

The longer the time control, the less difference DTZ will make. But at any time control there will be positions that SF+WDL will play suboptimally against SF+WDL+DTZ. Those will be positions with high DTZ value. But how often that happens in real play...

A position that will be won as white and (most likely) drawn as black by SF+WDL+DTZ:
[D]8/8/8/5N2/8/p7/8/2NK3k w - -

Another one:
[D]1R5Q/8/6k1/q7/8/8/8/1K6 b - -

A position that will be drawn as black but - perhaps - won as white by SF+WDL+DTZ (black does have some margin for error though, so SF+WDL might survive):
[D]K7/N7/k7/8/3p4/8/N7/8 w - -

Another one (with somewhat less margin):
[D]Q7/2k5/8/8/8/8/R2q4/1K6 b

[Ferdy]

Inspired by Kai's tests I was curious how SF8 plays with 5 piece syzygy and only WDL = rtbw-files. After 3000 selfplay games I got almost same +6 ELO as with full syzygy set! Previously I think engines with bitbases (wdl) have difficulties to benefit anything. Maybe SF is now different.Can You post just one 5 piece position, where SF8 cannot convert win without rtbz-file?

Tried it with long mate 5-men pos, with dm >= 10. Starting from longest mate, TC 60s + 0.2s inc/move, each opening is played twice.

Code: Select all

Score of Sf8-sy5 vs Sf8-wdl5: 46 - 22 - 29 [0.624]
Elo difference: 87.79 +/- 59.51

97 of 272 games finished.

5-men longest mate fen taken from,
http://kirill-kryukov.com/chess/longest ... ngth.shtml
and converted to epd with some opcodes. This is only for dm >= 10.

Code: Select all

8/8/8/8/1p2P3/4P3/1k6/3K4 w - - dm 127; c0 "KPPKP"; c1 "5-men";
8/4P3/7q/8/8/8/k1K1P3/8 w - - dm 124; c0 "KPPKQ"; c1 "5-men";
4Q3/8/7q/8/8/8/k3P3/2K5 w - - dm 124; c0 "KQPKQ"; c1 "5-men";
1N6/8/p7/8/8/8/2k1N3/K7 w - - dm 115; c0 "KNNKP"; c1 "5-men";
8/8/8/8/8/8/1n5B/2K1N2k w - - dm 107; c0 "KBNKN"; c1 "5-men";
8/8/8/8/3P2Q1/8/1p6/K1k5 w - - dm 105; c0 "KQPKP"; c1 "5-men";
8/8/3K4/8/8/1p6/k7/5B1N w - - dm 104; c0 "KBNKP"; c1 "5-men";
8/Q7/K7/5r2/5p2/8/6k1/8 w - - dm 104; c0 "KQKRP"; c1 "5-men";
K7/7P/4r3/8/5p2/8/2k5/8 w - - dm 103; c0 "KPKRP"; c1 "5-men";
1n6/3P4/8/8/1K6/7B/8/k7 w - - dm 100; c0 "KBPKN"; c1 "5-men";
k7/3P4/2n5/K7/8/8/3N4/8 w - - dm 97; c0 "KNPKN"; c1 "5-men";
8/8/7P/8/4k3/b7/8/1K3b2 w - - dm 83; c0 "KPKBB"; c1 "5-men";
8/1Q6/8/8/2bb4/8/3k4/K7 w - - dm 81; c0 "KQKBB"; c1 "5-men";
7B/6n1/8/6k1/6B1/8/8/K7 w - - dm 78; c0 "KBBKN"; c1 "5-men";
8/8/8/8/4k1B1/1p2B3/2K5/8 w - - dm 74; c0 "KBBKP"; c1 "5-men";
k7/8/8/8/8/4R3/6P1/1K5r w - - dm 74; c0 "KRPKR"; c1 "5-men";
8/8/7P/8/4n3/1k6/8/3K3n w - - dm 74; c0 "KPKNN"; c1 "5-men";
8/8/R7/8/8/1k1P4/6b1/K7 w - - dm 73; c0 "KRPKB"; c1 "5-men";
8/8/8/8/4n2Q/3k4/8/3K3n w - - dm 72; c0 "KQKNN"; c1 "5-men";
8/4P3/b7/r7/8/8/8/1K4k1 w - - dm 70; c0 "KPKRB"; c1 "5-men";
8/4Q3/b7/r7/8/8/8/K5k1 w - - dm 70; c0 "KQKRB"; c1 "5-men";
8/5Q2/8/n7/8/4k3/8/3K2r1 w - - dm 69; c0 "KQKRN"; c1 "5-men";
8/P1k5/8/8/8/1q6/R7/K7 w - - dm 68; c0 "KRPKQ"; c1 "5-men";
1n6/1K1P4/8/8/8/8/4r3/1k6 w - - dm 68; c0 "KPKRN"; c1 "5-men";
8/7p/8/8/8/1B6/K6P/2k5 w - - dm 67; c0 "KBPKP"; c1 "5-men";
8/8/8/8/q7/6k1/8/KR5Q w - - dm 67; c0 "KQRKQ"; c1 "5-men";
2K4k/7p/8/8/8/8/3R4/n7 w - - dm 67; c0 "KRKNP"; c1 "5-men";
8/3B4/8/1R6/5r2/8/3K4/5k2 w - - dm 65; c0 "KRBKR"; c1 "5-men";
8/3K4/8/6p1/P7/8/6n1/7k w - - dm 58; c0 "KPKNP"; c1 "5-men";
N7/8/K7/8/4p3/8/6P1/k7 w - - dm 57; c0 "KNPKP"; c1 "5-men";
R7/P7/8/K7/8/8/pk6/8 w - - dm 56; c0 "KRPKP"; c1 "5-men";
8/8/3P4/8/8/2k5/8/1K2n2b w - - dm 55; c0 "KPKBN"; c1 "5-men";
8/7Q/8/K7/8/8/3p4/n1k5 w - - dm 55; c0 "KQKNP"; c1 "5-men";
3K4/8/8/4P3/8/2r5/5P2/2k5 w - - dm 54; c0 "KPPKR"; c1 "5-men";
8/5n2/8/8/8/7k/7P/3K3R w - - dm 54; c0 "KRPKN"; c1 "5-men";
8/8/8/8/3b4/1Q6/3kn3/K7 w - - dm 53; c0 "KQKBN"; c1 "5-men";
8/8/6B1/8/2P5/8/2K5/k4b2 w - - dm 51; c0 "KBPKB"; c1 "5-men";
3K4/8/1P6/8/6p1/8/4b3/7k w - - dm 51; c0 "KPKBP"; c1 "5-men";
3n4/5P2/8/8/3K2P1/8/k7/8 w - - dm 50; c0 "KPPKN"; c1 "5-men";
3K4/4P3/1r2k3/8/8/8/8/7r w - - dm 50; c0 "KPKRR"; c1 "5-men";
3K4/8/8/8/8/1Q6/4b1pk/8 w - - dm 50; c0 "KQKBP"; c1 "5-men";
4r3/8/5Q2/8/5r2/1k6/8/1K6 w - - dm 49; c0 "KQKRR"; c1 "5-men";
8/r1k5/5P2/3K4/B7/8/8/8 w - - dm 45; c0 "KBPKR"; c1 "5-men";
1N6/8/8/PK6/3r4/3k4/8/8 w - - dm 44; c0 "KNPKR"; c1 "5-men";
8/6P1/7k/8/6P1/1K6/8/1b6 w - - dm 43; c0 "KPPKB"; c1 "5-men";
8/8/8/8/2K5/b7/1k5P/2N5 w - - dm 43; c0 "KNPKB"; c1 "5-men";
8/8/7P/3K4/5p2/4p3/8/7k w - - dm 43; c0 "KPKPP"; c1 "5-men";
8/2P2k2/3q4/2K2r2/8/8/8/8 w - - dm 43; c0 "KPKQR"; c1 "5-men";
8/4P3/8/N1K5/k7/8/7q/8 w - - dm 41; c0 "KNPKQ"; c1 "5-men";
6q1/8/8/Q7/8/2K5/8/3Nk3 w - - dm 41; c0 "KQNKQ"; c1 "5-men";
8/5R2/8/8/k7/4n3/3K1n2/8 w - - dm 41; c0 "KRKNN"; c1 "5-men";
8/5R2/8/8/k7/4b3/3K1n2/8 w - - dm 41; c0 "KRKBN"; c1 "5-men";
8/5R2/8/8/k7/4r3/3K1n2/8 w - - dm 41; c0 "KRKRN"; c1 "5-men";
8/5R2/8/8/k7/4q3/3K1n2/8 w - - dm 41; c0 "KRKQN"; c1 "5-men";
8/8/8/2K5/5p2/8/1Q2pk2/8 w - - dm 41; c0 "KQKPP"; c1 "5-men";
8/2R5/8/8/7k/4n3/6B1/3K4 w - - dm 40; c0 "KRBKN"; c1 "5-men";
8/2R5/8/8/7k/4n3/6R1/3K4 w - - dm 40; c0 "KRRKN"; c1 "5-men";
8/7B/8/7k/5Q2/8/8/Kr6 w - - dm 40; c0 "KQBKR"; c1 "5-men";
8/Q7/R7/8/8/8/1p6/K1k5 w - - dm 40; c0 "KQRKP"; c1 "5-men";
8/2R5/8/8/7k/4n3/6Q1/3K4 w - - dm 40; c0 "KQRKN"; c1 "5-men";
2K5/8/3pR3/5p2/7k/8/8/8 w - - dm 40; c0 "KRKPP"; c1 "5-men";
8/7B/8/4N3/8/8/3k4/K1b5 w - - dm 39; c0 "KBNKB"; c1 "5-men";
8/6bp/8/8/k1K5/8/8/2R5 w - - dm 39; c0 "KRKBP"; c1 "5-men";
k7/2Q5/8/8/8/8/3r4/1N1K4 w - - dm 38; c0 "KQNKR"; c1 "5-men";
8/8/6q1/8/8/8/2krQ3/K7 w - - dm 38; c0 "KQKQR"; c1 "5-men";
8/8/8/8/RN6/1k6/8/3K1n2 w - - dm 37; c0 "KRNKN"; c1 "5-men";
8/8/8/8/N2K4/R7/8/2k1r3 w - - dm 37; c0 "KRNKR"; c1 "5-men";
8/8/P7/3k4/r7/8/8/KQ6 w - - dm 37; c0 "KQPKR"; c1 "5-men";
8/8/8/2r5/3B4/8/3N4/K3k3 w - - dm 36; c0 "KBNKR"; c1 "5-men";
8/8/8/2q5/3B4/8/3N4/K3k3 w - - dm 36; c0 "KBNKQ"; c1 "5-men";
8/8/8/6Q1/5Q2/7k/r7/K7 w - - dm 35; c0 "KQQKR"; c1 "5-men";
8/3P1q2/2k5/6B1/8/8/8/K7 w - - dm 35; c0 "KBPKQ"; c1 "5-men";
8/8/8/6N1/6kB/6N1/8/K7 w - - dm 34; c0 "KBNNK"; c1 "5-men";
8/8/8/8/3RQ3/2k5/8/K4r2 w - - dm 34; c0 "KQRKR"; c1 "5-men";
k1q5/2K5/7p/8/8/8/6P1/8 w - - dm 34; c0 "KPKQP"; c1 "5-men";
K7/8/8/8/2k3P1/6P1/6P1/8 w - - dm 33; c0 "KPPPK"; c1 "5-men";
K1k5/2P5/8/7B/8/8/8/N7 w - - dm 33; c0 "KBNPK"; c1 "5-men";
8/8/8/8/8/B7/5Bk1/K6N w - - dm 33; c0 "KBBNK"; c1 "5-men";
8/8/6R1/8/8/8/4p1R1/3K1k2 w - - dm 33; c0 "KRRKP"; c1 "5-men";
7q/8/8/k7/8/7B/3K4/4Q3 w - - dm 33; c0 "KQBKQ"; c1 "5-men";
7R/8/8/k7/8/r7/Kp6/8 w - - dm 33; c0 "KRKRP"; c1 "5-men";
7R/8/8/k7/8/q7/Kp6/8 w - - dm 33; c0 "KRKQP"; c1 "5-men";
7N/K5k1/8/8/8/6P1/6P1/8 w - - dm 32; c0 "KNPPK"; c1 "5-men";
8/Q7/2K3k1/8/8/8/7p/2B5 w - - dm 32; c0 "KQBKP"; c1 "5-men";
8/8/8/8/4b3/8/2R5/3K1Nk1 w - - dm 31; c0 "KRNKB"; c1 "5-men";
k7/6R1/8/8/1R6/2r5/8/2K5 w - - dm 31; c0 "KRRKR"; c1 "5-men";
8/8/8/b1R5/8/3K4/8/1k4b1 w - - dm 31; c0 "KRKBB"; c1 "5-men";
4B3/6P1/7k/8/8/8/K7/1B6 w - - dm 30; c0 "KBBPK"; c1 "5-men";
8/8/8/8/8/3R4/6k1/3K1bB1 w - - dm 30; c0 "KRBKB"; c1 "5-men";
8/Q7/2n5/1k6/3K4/8/7P/8 w - - dm 30; c0 "KQPKN"; c1 "5-men";
8/8/2K4N/8/8/5Q2/p1k5/8 w - - dm 30; c0 "KQNKP"; c1 "5-men";
1Q6/Q7/8/8/8/8/2k5/K4q2 w - - dm 30; c0 "KQQKQ"; c1 "5-men";
8/8/8/8/Rr6/3K4/8/1k4b1 w - - dm 30; c0 "KRKRB"; c1 "5-men";
8/8/8/8/Rq6/3K4/8/1k4b1 w - - dm 30; c0 "KRKQB"; c1 "5-men";
8/8/8/8/8/8/6kB/K5NR w - - dm 29; c0 "KRBNK"; c1 "5-men";
8/8/8/4R3/8/2k3N1/3p4/K7 w - - dm 29; c0 "KRNKP"; c1 "5-men";
8/7R/1R6/8/4b3/8/2K5/5k2 w - - dm 29; c0 "KRRKB"; c1 "5-men";
k2q4/7R/8/8/8/8/8/2K3R1 w - - dm 29; c0 "KRRKQ"; c1 "5-men";
8/8/8/5b2/8/R2K4/6k1/1Q6 w - - dm 29; c0 "KQRKB"; c1 "5-men";
8/n7/8/1k1q4/3K3P/8/8/8 w - - dm 29; c0 "KPKQN"; c1 "5-men";
2Qq4/2K5/8/8/8/8/4p3/2k5 w - - dm 29; c0 "KQKQP"; c1 "5-men";
N1N5/8/8/8/8/4k2P/8/K7 w - - dm 28; c0 "KNNPK"; c1 "5-men";
8/8/8/8/8/1R1K4/kB1p4/8 w - - dm 28; c0 "KRBKP"; c1 "5-men";
8/8/k7/8/1Q6/2K5/6P1/4b3 w - - dm 28; c0 "KQPKB"; c1 "5-men";
B7/1K6/8/8/8/3k3P/7P/8 w - - dm 25; c0 "KBPPK"; c1 "5-men";
qk6/6P1/6b1/1K6/8/8/8/8 w - - dm 24; c0 "KPKQB"; c1 "5-men";
3q4/2n5/8/8/8/8/6Q1/k2K4 w - - dm 24; c0 "KQKQN"; c1 "5-men";
8/2b5/8/8/8/8/k2K1Q2/q7 w - - dm 24; c0 "KQKQB"; c1 "5-men";
4r3/8/8/8/8/8/8/k1K3BB w - - dm 23; c0 "KBBKR"; c1 "5-men";
8/8/8/2B5/8/8/3k4/K1b4B w - - dm 22; c0 "KBBKB"; c1 "5-men";
8/8/8/3Q4/7Q/2k5/1p6/K7 w - - dm 22; c0 "KQQKP"; c1 "5-men";
8/8/8/8/8/8/N4k2/KN4N1 w - - dm 21; c0 "KNNNK"; c1 "5-men";
8/8/8/1q4B1/8/3k4/6B1/K7 w - - dm 21; c0 "KBBKQ"; c1 "5-men";
1q6/8/8/8/8/8/8/k1KR2B1 w - - dm 21; c0 "KRBKQ"; c1 "5-men";
8/8/8/8/3N4/2k5/1n2Q3/3K4 w - - dm 21; c0 "KQNKN"; c1 "5-men";
8/8/8/8/8/2kB4/1n2Q3/3K4 w - - dm 21; c0 "KQBKN"; c1 "5-men";
8/8/8/8/R7/6q1/5N2/K1k5 w - - dm 20; c0 "KRNKQ"; c1 "5-men";
8/8/R2r4/8/8/3k4/1r6/K7 w - - dm 20; c0 "KRKRR"; c1 "5-men";
8/8/R2q4/8/8/3k4/1r6/K7 w - - dm 20; c0 "KRKQR"; c1 "5-men";
8/8/8/4k3/2Q5/n7/3Q4/1K6 w - - dm 19; c0 "KQQKN"; c1 "5-men";
8/8/8/8/8/8/1R4q1/Kqk5 w - - dm 19; c0 "KRKQQ"; c1 "5-men";
6k1/6NR/7P/8/8/8/8/K7 w - - dm 17; c0 "KRNPK"; c1 "5-men";
8/8/8/4b3/8/8/N1k5/K4Q2 w - - dm 17; c0 "KQNKB"; c1 "5-men";
8/8/8/4k3/3b4/8/5Q2/K3B3 w - - dm 17; c0 "KQBKB"; c1 "5-men";
8/p1p5/8/8/8/8/8/kNK5 w - - dm 17; c0 "KNKPP"; c1 "5-men";
8/8/8/3B4/3B4/3k4/3B4/3K4 w - - dm 16; c0 "KBBBK"; c1 "5-men";
8/8/8/8/BR6/kP6/8/K7 w - - dm 16; c0 "KRBPK"; c1 "5-men";
8/8/8/8/2kP4/8/1R1P4/K7 w - - dm 15; c0 "KRPPK"; c1 "5-men";
8/5NR1/5kN1/8/8/8/8/K7 w - - dm 15; c0 "KRNNK"; c1 "5-men";
8/8/8/8/8/3k1b2/8/2QKQ3 w - - dm 15; c0 "KQQKB"; c1 "5-men";
8/K7/8/8/5k2/4RPR1/8/8 w - - dm 14; c0 "KRRPK"; c1 "5-men";
qk4q1/5P2/8/1K6/8/8/8/8 w - - dm 13; c0 "KPKQQ"; c1 "5-men";
8/8/8/8/8/8/3q4/qk1K2Q1 w - - dm 13; c0 "KQKQQ"; c1 "5-men";
8/8/8/8/3B4/3k4/2RB4/K7 w - - dm 12; c0 "KRBBK"; c1 "5-men";
8/8/6R1/6Nk/6R1/8/8/K7 w - - dm 10; c0 "KRRNK"; c1 "5-men";
8/6R1/6Bk/6R1/8/8/8/K7 w - - dm 10; c0 "KRRBK"; c1 "5-men";

Sample failed to win pos.

[d]8/8/8/8/1p2P3/4P3/1k6/3K4 w - - 0 1
[d]1N6/8/p7/8/8/8/2k1N3/K7 w - - 0 1
[d]8/8/8/8/8/8/1n5B/2K1N2k w - - 0 1
[d]8/8/8/8/3P2Q1/8/1p6/K1k5 w - - 0 1
[d]8/8/3K4/8/8/1p6/k7/5B1N w - - 0 1

[yurikvelo]

Inspired by Kai's tests I was curious how SF8 plays with 5 piece syzygy and only WDL = rtbw-files. After 3000 selfplay games I got almost same +6 ELO as with full syzygy set! Previously I think engines with bitbases (wdl) have difficulties to benefit anything. Maybe SF is now different.Can You post just one 5 piece position, where SF8 cannot convert win without rtbz-file?

Did you adjudicate games at +6.5?

[Laskos]

Inspired by Kai's tests I was curious how SF8 plays with 5 piece syzygy and only WDL = rtbw-files. After 3000 selfplay games I got almost same +6 ELO as with full syzygy set! Previously I think engines with bitbases (wdl) have difficulties to benefit anything. Maybe SF is now different.Can You post just one 5 piece position, where SF8 cannot convert win without rtbz-file?

Well, on hard 5-men positions (I got them from regular 5-men by using Syzygy and the filtering for longest mates), I got the following:

Code: Select all

Games Completed = 200 of 200 (Avg game length = 32.120 sec)
Settings = Gauntlet/64MB/400ms per move/M 600000cp for 3000 moves, D 120000 moves/EPD:C:\LittleBlitzer\hard5men.epd(258)
Time = 1633 sec elapsed, 0 sec remaining
 1.  SF Syzygy WDL+DTZ        	132.5/200	100-35-65  	(L: m=35 t=0 i=0 a=0)	(D: r=6 i=0 f=59 s=0 a=0)	(tpm=385.1 d=42.39 nps=2622583)
 2.  SF Syzygy WDL            	 67.5/200	35-100-65  	(L: m=100 t=0 i=0 a=0)	(D: r=6 i=0 f=59 s=0 a=0)	(tpm=380.6 d=44.99 nps=2570182)

WDL alone performs miserably and solves only 35/100 of them.

But from regular (somewhat unbalanced) endgames, there is no sensible difference:

Code: Select all

Games Completed = 3000 of 3000 (Avg game length = 9.625 sec)
Settings = RR/64MB/100ms per move/M 600000cp for 1000 moves, D 120000 moves/EPD:C:\LittleBlitzer\END_789_13_17.epd(897)
Time = 7558 sec elapsed, 0 sec remaining
 1.  SF Syzygy WDL+DTZ        	1030.0/2000	593-533-874  	(L: m=533 t=0 i=0 a=0)	(D: r=337 i=283 f=223 s=31 a=0)	(tpm=107.0 d=30.65 nps=2684922)
 2.  SF Syzygy WDL            	1025.5/2000	603-552-845  	(L: m=552 t=0 i=0 a=0)	(D: r=322 i=268 f=222 s=33 a=0)	(tpm=106.8 d=31.61 nps=2609905)
 3.  SF No Syzygy             	 944.5/2000	516-627-857  	(L: m=627 t=0 i=0 a=0)	(D: r=309 i=287 f=229 s=32 a=0)	(tpm=107.0 d=29.27 nps=2812420)

So, probably for analysis it's a big deal, for gameplay just a few Elo points difference using WDL or WDL+DTZ.

[Dann Corbit]

When there is a clearly winning position, the engine will almost always win even without the tablebase files.

The times that is loses are not enough to make a noticeable Elo difference.

The main difference with having them is:
1. The game will be prettier.
2. If you are doing analysis, you will get the right answer.

[Jouni]

Thanks for replies. Obviously SF plays great with wdl only - note it reaches max depth 127 easily in 5 piece endings (not with full syzy)! Also Komodo is good with wdl, but Houdini cannot get any benefit from them - buggy implemention?

[Jouni]

Actually Houdini plays clearly worse with WDL only:

Code: Select all

Games Completed = 100 of 1000 (Avg game length = 34.336 sec)
Settings = Gauntlet/64MB/10000ms+100ms/M 99999cp for 300 moves, D 300 moves/EPD:endings.epd(1)
Time = 875 sec elapsed, 7874 sec remaining
 1.  HOU no TB                	61.0/100	30-8-62  	(L: m=6 t=2 i=0 a=0)	(D: r=52 i=6 f=3 s=1 a=0)	(tpm=214.8 d=15.16 nps=2640492)
 2.  HOU WDL                  	39.0/100	8-30-62  	(L: m=28 t=2 i=0 a=0)	(D: r=52 i=6 f=3 s=1 a=0)	(tpm=214.9 d=14.14 nps=2460874)
 

No wonder I didn't measure any gain even with all 5 piece files .

[Jouni]

Stockfish 8 draws almost all long Mosca positions with DTZ file also against Komodo. Why??

[syzygy]

Stockfish 8 draws almost all long Mosca positions with DTZ file also against Komodo. Why??

Positions with dm > 50 may be 50-move draws. If black plays those well, white cannot win them.