I just stopped a test which ran non-stop from March 23rd to see the effect of 6-men Syzygy tablebases on Komodo 10.4. Conditions:mjlef wrote:You actually need a lot of games to reduce error margins enough. 3000 game is roughly +/- 7.7 elo. 10000 games is roughly +/- 4.4 elo. In the past we showed something like +10 elo for 5 piece Syzygy and more fr 6 piece. But we have changed some endgame eval rules in Komodo since running this. So perhaps Syzygy help less now. But I cannot say due to the larger error margins. Most of our tests have been using SSD which is faster, although it should not matter too much for 5 piece games since they quickly get cached to RAM by most operating systems (the working win-loss-draw set is only around 500 MB).Jouni wrote:I re-run test (3*3000) games with different book and slightly modified settings. Syzygy files on HD. Now I got this TB gain:
SF8 +9,2
Houdini5 +6,5
Komodo10 +0,7(!?)
I You look at average depth You see, that Komodo with TBs has lower than without, but SF has bigger depth with TBs. For Komodo You obviously need some 10k games to see any statistical gain.
BTW Komodo has a "Smart Syzygy" function for people with HD instead of SSD. When turned on it always looks up 5 piece positions, but only looks up 6 piece positions at the depth the user sets. 6 piece are very slow to access on a HD, so this feature gives a reasonable mix of accuracy versus speed. If you ever try a 6 piece run, I would like to hear how this feature performs on your HD system.
Thanks for running these.
• 4-core i7-2600 3.4GHz, HT off (bios), Win 10
• Complete 6-men Syzygy on Samsung SSD 840 (WDL), and local SATA drive (DTZ)
• Komodo 10.4, default settings except: variety = 15
• Opponent: Komodo 10.4 no-syzygy, same settings but “Use Syzygy” was left unchecked
• Hash 512Mb per engine (system memory 8Gb)
• Fritz GUI v.14, draw = never, resign = never
• Time control: game in 3 min + 2 sec/move (average game almost exactly 10 min.)
• Opening book: Perfect2015s (Sedat); each opening played by both sides
Result:
Komodo 10.4 64-bit +8 +282/=2679/-205 51.22% 1621.5/3166
Komodo 10.4 noSyz -8 +205/=2679/-282 48.78% 1544.5/3166
Comments:
• Komodo is stable (no crashes)
• I was pleased with the book, as it mixed things up well. In conjunction with the variety=15 setting it resulted in zero doubles. There were still some short games, but only 167 were under 30 moves.
• I took a quick look at the subset of 3,166 games where at some point material was reduced to 8 men. There were 1,589 such games, and the impact of Syzygy on this subset was understandably higher:
Komodo 10.4 64-bit +21 +275/=1134/-180 52.99% 842.0/1589
Komodo 10.4 noSyz -21 +180/=1134/-275 47.01% 747.0/1589
Regards,
PK
