The Tablebase fixes are already in the release 18.1 which will be used for TCEC cup and CCCC2.
Also the "minimaxing back-up of the values" is already in the works, with the name of "certainty propagation".
Lc0 v18 rc2 for 2080 Syzygy implementation
Moderators: hgm, Rebel, chrisw
-
- Posts: 37
- Joined: Wed Feb 10, 2016 6:20 am
- Location: Singapore
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0 v18 rc2 for 2080 Syzygy implementation
Now, the really great news for Lc0 18.1 Syzygy: from regular openings, it improves substantially due to Syzygy 6-men (WDL + DTZ) on SSD, more than regular engines. It is somehow unexpected to me, as from positions close to the TBs at root, it improves similarly to regular engines. I used Adam Hair's 4-mover PGN openings of pretty regular openings. Time control is short, 0.2s/move. I expect that at longer time control, the result might be even more impressive, as more games will be decided in later stages of the games. I played a gauntlet of 2000 games total, SF8_Syzygy against Lc0 18.1 ID11261 with and without Syzygy. GPU is GTX 1060, CPU is strong 4 core i7. Here is the result:
A whopping 47 Elo points improvement (47 +/- 13 with 68% confidence) due to Syzygy 6-men. This is about 3 times larger improvement than what regular engines get from Syzygy from regular openings. In these conditions, in CCCC division with 8 engines, Lc0 might end in top 3, overcoming one of the uber-top 3. Probably not in top 2, though. If CCCC had a round-robin of only top 4 engines (and not 8), Lc0 had a good shot at entering top 2 and the final, because it performs better against stronger engines.
Code: Select all
Rank Name Elo +/- Games Score Draws
0 SF8_Syzygy 150 13 2000 70.3% 31.4%
1 lc0_v18.1_Syzygy -127 18 1000 32.5% 33.0%
2 lc0_v18.1 -174 19 1000 26.9% 29.8%
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0 v18 rc2 for 2080 Syzygy implementation
That was against a stronger engine in these conditions, SF8. But Lc0 performs pretty well against stronger engines. Where it underperforms is against weaker engines, it often happens that it fails to convert very favorable middlegames or won endgames. I was curious if v18.1 Syzygy (6-men on SSD) cures a bit of this problem against weaker engines.Laskos wrote: ↑Thu Oct 04, 2018 8:10 am Now, the really great news for Lc0 18.1 Syzygy: from regular openings, it improves substantially due to Syzygy 6-men (WDL + DTZ) on SSD, more than regular engines. It is somehow unexpected to me, as from positions close to the TBs at root, it improves similarly to regular engines. I used Adam Hair's 4-mover PGN openings of pretty regular openings. Time control is short, 0.2s/move. I expect that at longer time control, the result might be even more impressive, as more games will be decided in later stages of the games. I played a gauntlet of 2000 games total, SF8_Syzygy against Lc0 18.1 ID11261 with and without Syzygy. GPU is GTX 1060, CPU is strong 4 core i7. Here is the result:
A whopping 47 Elo points improvement (47 +/- 13 with 68% confidence) due to Syzygy 6-men. This is about 3 times larger improvement than what regular engines get from Syzygy from regular openings. In these conditions, in CCCC division with 8 engines, Lc0 might end in top 3, overcoming one of the uber-top 3. Probably not in top 2, though. If CCCC had a round-robin of only top 4 engines (and not 8), Lc0 had a good shot at entering top 2 and the final, because it performs better against stronger engines.Code: Select all
Rank Name Elo +/- Games Score Draws 0 SF8_Syzygy 150 13 2000 70.3% 31.4% 1 lc0_v18.1_Syzygy -127 18 1000 32.5% 33.0% 2 lc0_v18.1 -174 19 1000 26.9% 29.8%
No adjudications. GPU --- GTX 1060, CPU --- 4 core i7
The first gauntlet, 1000 games at 0.2s/move, against the weaker Arasan 20.5:
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 -135 20 1000 31.4% 25.1%
1 lc0_v18.1_Syzygy 162 29 500 71.8% 22.4%
2 lc0_v17 110 27 500 65.3% 27.8%
The second gauntlet, 200 games at 1.0s/move, against the same weaker Arasan 20.5:
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 -295 52 200 15.5% 21.0%
1 lc0_v18.1_Syzygy 328 83 100 86.8% 16.5%
2 lc0_v17 265 69 100 82.2% 25.5%
Finished match
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0 v18 rc2 for 2080 Syzygy implementation
And this is not so good news: Lc0 plays significantly worse in the endgames than openings and midgames, and scales badly in endgames to longer TC, no better than regular AB engines. In general play from short openings in my conditions, it scales significantly better.Laskos wrote: ↑Tue Oct 09, 2018 11:45 amThat was against a stronger engine in these conditions, SF8. But Lc0 performs pretty well against stronger engines. Where it underperforms is against weaker engines, it often happens that it fails to convert very favorable middlegames or won endgames. I was curious if v18.1 Syzygy (6-men on SSD) cures a bit of this problem against weaker engines.Laskos wrote: ↑Thu Oct 04, 2018 8:10 am Now, the really great news for Lc0 18.1 Syzygy: from regular openings, it improves substantially due to Syzygy 6-men (WDL + DTZ) on SSD, more than regular engines. It is somehow unexpected to me, as from positions close to the TBs at root, it improves similarly to regular engines. I used Adam Hair's 4-mover PGN openings of pretty regular openings. Time control is short, 0.2s/move. I expect that at longer time control, the result might be even more impressive, as more games will be decided in later stages of the games. I played a gauntlet of 2000 games total, SF8_Syzygy against Lc0 18.1 ID11261 with and without Syzygy. GPU is GTX 1060, CPU is strong 4 core i7. Here is the result:
A whopping 47 Elo points improvement (47 +/- 13 with 68% confidence) due to Syzygy 6-men. This is about 3 times larger improvement than what regular engines get from Syzygy from regular openings. In these conditions, in CCCC division with 8 engines, Lc0 might end in top 3, overcoming one of the uber-top 3. Probably not in top 2, though. If CCCC had a round-robin of only top 4 engines (and not 8), Lc0 had a good shot at entering top 2 and the final, because it performs better against stronger engines.Code: Select all
Rank Name Elo +/- Games Score Draws 0 SF8_Syzygy 150 13 2000 70.3% 31.4% 1 lc0_v18.1_Syzygy -127 18 1000 32.5% 33.0% 2 lc0_v18.1 -174 19 1000 26.9% 29.8%
No adjudications. GPU --- GTX 1060, CPU --- 4 core i7
The first gauntlet, 1000 games at 0.2s/move, against the weaker Arasan 20.5:
+52 Elo points due to Syzygy, with decreasing draw rate when using Syzygy, probably converting won endgames to actual winsCode: Select all
Rank Name Elo +/- Games Score Draws 0 Arasan205 -135 20 1000 31.4% 25.1% 1 lc0_v18.1_Syzygy 162 29 500 71.8% 22.4% 2 lc0_v17 110 27 500 65.3% 27.8%
The second gauntlet, 200 games at 1.0s/move, against the same weaker Arasan 20.5:
+63 Elo pints due to Syzygy, but the error margins here are large. Now the draw rate decreased dramatically using Syzygy, while losses rate is pretty much the same, but win rate goes much higher. It means that some (many) games which non-Syzygy version was only able to draw, were actually won by Syzygy Lc0. Probably the usual occurrence with Lc0, not managing to win very advantageous endgames, is somewhat cured with Syzygy usage. I am curious whether in CCCC2, Lc0 will perform better against weaker engines, managing to be less drawish against them.Code: Select all
Rank Name Elo +/- Games Score Draws 0 Arasan205 -295 52 200 15.5% 21.0% 1 lc0_v18.1_Syzygy 328 83 100 86.8% 16.5% 2 lc0_v17 265 69 100 82.2% 25.5% Finished match
A disbalanced late endgame suite, most of the positions in it being won/lost positions. With these disbalanced suites playing side-and-reversed, pentanomial (correct) error margins are 2.0-2.5 smaller than those shown in cutechess-cli (trinomial error margins), so I was able to reduce significantly the erro margins and enhance the significance of the result playing the same number of games.
ID11261
0.2s/move:
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 50 20 1000 57.1% 17.8%
1 lc0_v18_Syzygy -23 28 500 46.7% 16.6%
2 lc0_v17 -77 28 500 39.1% 19.0%
Finished match
1.0s/move
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 63 45 200 59.0% 17.0%
1 lc0_v18_Syzygy -31 64 100 45.5% 15.0%
2 lc0_v17 -96 64 100 36.5% 19.0%
Finished match
Although the result still shows an increasing with TC importance of Syzygy, but the absolute performance in late endgames against Arasan is even worse at longer time control than at shorter one. And compare the endgame performance, where Arasan is 31 Elo points stronger than Lc0 v18.1 Syzygy at 1.0s/move, while in general play:
Code: Select all
Rank Name Elo +/- Games Score Draws
0 Arasan205 -295 52 200 15.5% 21.0%
1 lc0_v18.1_Syzygy 328 83 100 86.8% 16.5%
2 lc0_v17 265 69 100 82.2% 25.5%
Finished match
-
- Posts: 204
- Joined: Tue Oct 15, 2013 2:34 am
- Location: US
- Full name: Mike Babigian
Re: Lc0 v18 rc2 for 2080 Syzygy implementation
The solution may be simpler than most think. NN's can only learn so much before saturation and the current preferred solution is to make the nets bigger. This however slows performance. Instead of using bigger nets, use more of them.
Train a net from the starting position as we do now, but train another net on the same training data starting after 4 men have been captured. Train another starting after 8 men have been removed and so on until you have 7 nets. When the engine is playing (not training), automatically switch nets when the number of pieces on the board change. This allows each net to learn more about that game stage without making it larger and slower.
This is the same thing that happens with Syzygy tables. When the piece count drops from 7 pieces to 6, the engines switch to a different set of files.
Theoretically the nets could be smaller the closer you get to the endgame. I.e. 256x20 may be overkill to learn 4 men endgames.
My 2 cents.
P.S. Switching nets in search would probably be too slow. You'd likely only switch nets when the current board position loses a piece.
Train a net from the starting position as we do now, but train another net on the same training data starting after 4 men have been captured. Train another starting after 8 men have been removed and so on until you have 7 nets. When the engine is playing (not training), automatically switch nets when the number of pieces on the board change. This allows each net to learn more about that game stage without making it larger and slower.
This is the same thing that happens with Syzygy tables. When the piece count drops from 7 pieces to 6, the engines switch to a different set of files.
Theoretically the nets could be smaller the closer you get to the endgame. I.e. 256x20 may be overkill to learn 4 men endgames.
My 2 cents.
P.S. Switching nets in search would probably be too slow. You'd likely only switch nets when the current board position loses a piece.
“Censorship is telling a man he can't have a steak just because a baby can't chew it.” ― Mark Twain