Lc0 v18 rc2 for 2080 Syzygy implementation

[megamau]

The Tablebase fixes are already in the release 18.1 which will be used for TCEC cup and CCCC2.

Also the "minimaxing back-up of the values" is already in the works, with the name of "certainty propagation".

[Laskos]

Now, the really great news for Lc0 18.1 Syzygy: from regular openings, it improves substantially due to Syzygy 6-men (WDL + DTZ) on SSD, more than regular engines. It is somehow unexpected to me, as from positions close to the TBs at root, it improves similarly to regular engines. I used Adam Hair's 4-mover PGN openings of pretty regular openings. Time control is short, 0.2s/move. I expect that at longer time control, the result might be even more impressive, as more games will be decided in later stages of the games. I played a gauntlet of 2000 games total, SF8_Syzygy against Lc0 18.1 ID11261 with and without Syzygy. GPU is GTX 1060, CPU is strong 4 core i7. Here is the result:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 SF8_Syzygy                    150      13    2000   70.3%   31.4%
   
   1 lc0_v18.1_Syzygy             -127      18    1000   32.5%   33.0%
   2 lc0_v18.1                    -174      19    1000   26.9%   29.8% 

A whopping 47 Elo points improvement (47 +/- 13 with 68% confidence) due to Syzygy 6-men. This is about 3 times larger improvement than what regular engines get from Syzygy from regular openings. In these conditions, in CCCC division with 8 engines, Lc0 might end in top 3, overcoming one of the uber-top 3. Probably not in top 2, though. If CCCC had a round-robin of only top 4 engines (and not 8), Lc0 had a good shot at entering top 2 and the final, because it performs better against stronger engines.

[Laskos]

Now, the really great news for Lc0 18.1 Syzygy: from regular openings, it improves substantially due to Syzygy 6-men (WDL + DTZ) on SSD, more than regular engines. It is somehow unexpected to me, as from positions close to the TBs at root, it improves similarly to regular engines. I used Adam Hair's 4-mover PGN openings of pretty regular openings. Time control is short, 0.2s/move. I expect that at longer time control, the result might be even more impressive, as more games will be decided in later stages of the games. I played a gauntlet of 2000 games total, SF8_Syzygy against Lc0 18.1 ID11261 with and without Syzygy. GPU is GTX 1060, CPU is strong 4 core i7. Here is the result:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 SF8_Syzygy                    150      13    2000   70.3%   31.4%
   
   1 lc0_v18.1_Syzygy             -127      18    1000   32.5%   33.0%
   2 lc0_v18.1                    -174      19    1000   26.9%   29.8% 

A whopping 47 Elo points improvement (47 +/- 13 with 68% confidence) due to Syzygy 6-men. This is about 3 times larger improvement than what regular engines get from Syzygy from regular openings. In these conditions, in CCCC division with 8 engines, Lc0 might end in top 3, overcoming one of the uber-top 3. Probably not in top 2, though. If CCCC had a round-robin of only top 4 engines (and not 8), Lc0 had a good shot at entering top 2 and the final, because it performs better against stronger engines.

That was against a stronger engine in these conditions, SF8. But Lc0 performs pretty well against stronger engines. Where it underperforms is against weaker engines, it often happens that it fails to convert very favorable middlegames or won endgames. I was curious if v18.1 Syzygy (6-men on SSD) cures a bit of this problem against weaker engines.

No adjudications. GPU --- GTX 1060, CPU --- 4 core i7

The first gauntlet, 1000 games at 0.2s/move, against the weaker Arasan 20.5:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 Arasan205                    -135      20    1000   31.4%   25.1%
   
   1 lc0_v18.1_Syzygy              162      29     500   71.8%   22.4%
   2 lc0_v17                       110      27     500   65.3%   27.8%

+52 Elo points due to Syzygy, with decreasing draw rate when using Syzygy, probably converting won endgames to actual wins

The second gauntlet, 200 games at 1.0s/move, against the same weaker Arasan 20.5:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 Arasan205                    -295      52     200   15.5%   21.0%
   
   1 lc0_v18.1_Syzygy              328      83     100   86.8%   16.5%
   2 lc0_v17                       265      69     100   82.2%   25.5%
Finished match

+63 Elo pints due to Syzygy, but the error margins here are large. Now the draw rate decreased dramatically using Syzygy, while losses rate is pretty much the same, but win rate goes much higher. It means that some (many) games which non-Syzygy version was only able to draw, were actually won by Syzygy Lc0. Probably the usual occurrence with Lc0, not managing to win very advantageous endgames, is somewhat cured with Syzygy usage. I am curious whether in CCCC2, Lc0 will perform better against weaker engines, managing to be less drawish against them.

[Laskos]

Now, the really great news for Lc0 18.1 Syzygy: from regular openings, it improves substantially due to Syzygy 6-men (WDL + DTZ) on SSD, more than regular engines. It is somehow unexpected to me, as from positions close to the TBs at root, it improves similarly to regular engines. I used Adam Hair's 4-mover PGN openings of pretty regular openings. Time control is short, 0.2s/move. I expect that at longer time control, the result might be even more impressive, as more games will be decided in later stages of the games. I played a gauntlet of 2000 games total, SF8_Syzygy against Lc0 18.1 ID11261 with and without Syzygy. GPU is GTX 1060, CPU is strong 4 core i7. Here is the result:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 SF8_Syzygy                    150      13    2000   70.3%   31.4%
   
   1 lc0_v18.1_Syzygy             -127      18    1000   32.5%   33.0%
   2 lc0_v18.1                    -174      19    1000   26.9%   29.8% 

A whopping 47 Elo points improvement (47 +/- 13 with 68% confidence) due to Syzygy 6-men. This is about 3 times larger improvement than what regular engines get from Syzygy from regular openings. In these conditions, in CCCC division with 8 engines, Lc0 might end in top 3, overcoming one of the uber-top 3. Probably not in top 2, though. If CCCC had a round-robin of only top 4 engines (and not 8), Lc0 had a good shot at entering top 2 and the final, because it performs better against stronger engines.

That was against a stronger engine in these conditions, SF8. But Lc0 performs pretty well against stronger engines. Where it underperforms is against weaker engines, it often happens that it fails to convert very favorable middlegames or won endgames. I was curious if v18.1 Syzygy (6-men on SSD) cures a bit of this problem against weaker engines.

No adjudications. GPU --- GTX 1060, CPU --- 4 core i7

The first gauntlet, 1000 games at 0.2s/move, against the weaker Arasan 20.5:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 Arasan205                    -135      20    1000   31.4%   25.1%
   
   1 lc0_v18.1_Syzygy              162      29     500   71.8%   22.4%
   2 lc0_v17                       110      27     500   65.3%   27.8%

+52 Elo points due to Syzygy, with decreasing draw rate when using Syzygy, probably converting won endgames to actual wins

The second gauntlet, 200 games at 1.0s/move, against the same weaker Arasan 20.5:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 Arasan205                    -295      52     200   15.5%   21.0%
   
   1 lc0_v18.1_Syzygy              328      83     100   86.8%   16.5%
   2 lc0_v17                       265      69     100   82.2%   25.5%
Finished match

+63 Elo pints due to Syzygy, but the error margins here are large. Now the draw rate decreased dramatically using Syzygy, while losses rate is pretty much the same, but win rate goes much higher. It means that some (many) games which non-Syzygy version was only able to draw, were actually won by Syzygy Lc0. Probably the usual occurrence with Lc0, not managing to win very advantageous endgames, is somewhat cured with Syzygy usage. I am curious whether in CCCC2, Lc0 will perform better against weaker engines, managing to be less drawish against them.

And this is not so good news: Lc0 plays significantly worse in the endgames than openings and midgames, and scales badly in endgames to longer TC, no better than regular AB engines. In general play from short openings in my conditions, it scales significantly better.

A disbalanced late endgame suite, most of the positions in it being won/lost positions. With these disbalanced suites playing side-and-reversed, pentanomial (correct) error margins are 2.0-2.5 smaller than those shown in cutechess-cli (trinomial error margins), so I was able to reduce significantly the erro margins and enhance the significance of the result playing the same number of games.

ID11261

0.2s/move:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 Arasan205                      50      20    1000   57.1%   17.8%
   
   1 lc0_v18_Syzygy                -23      28     500   46.7%   16.6%
   2 lc0_v17                       -77      28     500   39.1%   19.0%
Finished match

+54 Elo points due to Syzygy.



1.0s/move

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 Arasan205                      63      45     200   59.0%   17.0%
   
   1 lc0_v18_Syzygy                -31      64     100   45.5%   15.0%
   2 lc0_v17                       -96      64     100   36.5%   19.0%
Finished match

+65 Elo points due to Syzygy.

Although the result still shows an increasing with TC importance of Syzygy, but the absolute performance in late endgames against Arasan is even worse at longer time control than at shorter one. And compare the endgame performance, where Arasan is 31 Elo points stronger than Lc0 v18.1 Syzygy at 1.0s/move, while in general play:

Code: Select all

Rank Name                          Elo     +/-   Games   Score   Draws
   0 Arasan205                    -295      52     200   15.5%   21.0%
   
   1 lc0_v18.1_Syzygy              328      83     100   86.8%   16.5%
   2 lc0_v17                       265      69     100   82.2%   25.5%
Finished match

Arasan is 328 Elo points weaker than Lc0 v18.1 Syzygy at 1.0s/move. So, in endgames, Lc0 underperforms by a whopping 360 Elo points compared to general play, and Syzygy helps just a bit. Really, something has to be done with endgames to start being competitive. Maybe TB rescoring will work in learning, but I am skeptical.

[mbabigian]

The solution may be simpler than most think. NN's can only learn so much before saturation and the current preferred solution is to make the nets bigger. This however slows performance. Instead of using bigger nets, use more of them.

Train a net from the starting position as we do now, but train another net on the same training data starting after 4 men have been captured. Train another starting after 8 men have been removed and so on until you have 7 nets. When the engine is playing (not training), automatically switch nets when the number of pieces on the board change. This allows each net to learn more about that game stage without making it larger and slower.

This is the same thing that happens with Syzygy tables. When the piece count drops from 7 pieces to 6, the engines switch to a different set of files.

Theoretically the nets could be smaller the closer you get to the endgame. I.e. 256x20 may be overkill to learn 4 men endgames.

My 2 cents.

P.S. Switching nets in search would probably be too slow. You'd likely only switch nets when the current board position loses a piece.