And this is not so good news: Lc0 plays significantly worse in the endgames than openings and midgames, and scales badly in endgames to longer TC, no better than regular AB engines. In general play from short openings in my conditions, it scales significantly better.Laskos wrote: ↑Tue Oct 09, 2018 11:45 amThat was against a stronger engine in these conditions, SF8. But Lc0 performs pretty well against stronger engines. Where it underperforms is against weaker engines, it often happens that it fails to convert very favorable middlegames or won endgames. I was curious if v18.1 Syzygy (6-men on SSD) cures a bit of this problem against weaker engines.Laskos wrote: ↑Thu Oct 04, 2018 8:10 am Now, the really great news for Lc0 18.1 Syzygy: from regular openings, it improves substantially due to Syzygy 6-men (WDL + DTZ) on SSD, more than regular engines. It is somehow unexpected to me, as from positions close to the TBs at root, it improves similarly to regular engines. I used Adam Hair's 4-mover PGN openings of pretty regular openings. Time control is short, 0.2s/move. I expect that at longer time control, the result might be even more impressive, as more games will be decided in later stages of the games. I played a gauntlet of 2000 games total, SF8_Syzygy against Lc0 18.1 ID11261 with and without Syzygy. GPU is GTX 1060, CPU is strong 4 core i7. Here is the result:
A whopping 47 Elo points improvement (47 +/- 13 with 68% confidence) due to Syzygy 6-men. This is about 3 times larger improvement than what regular engines get from Syzygy from regular openings. In these conditions, in CCCC division with 8 engines, Lc0 might end in top 3, overcoming one of the uber-top 3. Probably not in top 2, though. If CCCC had a round-robin of only top 4 engines (and not 8), Lc0 had a good shot at entering top 2 and the final, because it performs better against stronger engines.Code: Select all
Rank Name Elo +/- Games Score Draws 0 SF8_Syzygy 150 13 2000 70.3% 31.4% 1 lc0_v18.1_Syzygy -127 18 1000 32.5% 33.0% 2 lc0_v18.1 -174 19 1000 26.9% 29.8%
No adjudications. GPU --- GTX 1060, CPU --- 4 core i7
The first gauntlet, 1000 games at 0.2s/move, against the weaker Arasan 20.5:
+52 Elo points due to Syzygy, with decreasing draw rate when using Syzygy, probably converting won endgames to actual winsCode: Select all
Rank Name Elo +/- Games Score Draws 0 Arasan205 -135 20 1000 31.4% 25.1% 1 lc0_v18.1_Syzygy 162 29 500 71.8% 22.4% 2 lc0_v17 110 27 500 65.3% 27.8%
The second gauntlet, 200 games at 1.0s/move, against the same weaker Arasan 20.5:
+63 Elo pints due to Syzygy, but the error margins here are large. Now the draw rate decreased dramatically using Syzygy, while losses rate is pretty much the same, but win rate goes much higher. It means that some (many) games which non-Syzygy version was only able to draw, were actually won by Syzygy Lc0. Probably the usual occurrence with Lc0, not managing to win very advantageous endgames, is somewhat cured with Syzygy usage. I am curious whether in CCCC2, Lc0 will perform better against weaker engines, managing to be less drawish against them.Code: Select all
Rank Name Elo +/- Games Score Draws 0 Arasan205 -295 52 200 15.5% 21.0% 1 lc0_v18.1_Syzygy 328 83 100 86.8% 16.5% 2 lc0_v17 265 69 100 82.2% 25.5% Finished match
A disbalanced late endgame suite, most of the positions in it being won/lost positions. With these disbalanced suites playing side-and-reversed, pentanomial (correct) error margins are 2.0-2.5 smaller than those shown in cutechess-cli (trinomial error margins), so I was able to reduce significantly the erro margins and enhance the significance of the result playing the same number of games.
ID11261
0.2s/move:
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 50 20 1000 57.1% 17.8%
1 lc0_v18_Syzygy -23 28 500 46.7% 16.6%
2 lc0_v17 -77 28 500 39.1% 19.0%
Finished match
1.0s/move
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 63 45 200 59.0% 17.0%
1 lc0_v18_Syzygy -31 64 100 45.5% 15.0%
2 lc0_v17 -96 64 100 36.5% 19.0%
Finished match
Although the result still shows an increasing with TC importance of Syzygy, but the absolute performance in late endgames against Arasan is even worse at longer time control than at shorter one. And compare the endgame performance, where Arasan is 31 Elo points stronger than Lc0 v18.1 Syzygy at 1.0s/move, while in general play:
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 -295 52 200 15.5% 21.0%
1 lc0_v18.1_Syzygy 328 83 100 86.8% 16.5%
2 lc0_v17 265 69 100 82.2% 25.5%
Finished match