LCZero Accomplishments and Goals Thus Far

Discussion of anything and everything relating to chess playing software and machines.

Moderators: Harvey Williamson, bob, hgm

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 10839
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: LCZero Accomplishments and Goals Thus Far

Post by Laskos » Wed May 02, 2018 2:10 pm

Milos wrote:
Laskos wrote:
Albert Silver wrote:
Daniel Shawul wrote:It still blunders hugely after the +300 elo.
It does, no question about it. Even with the new cuDNN speedups (which need testing to see how that translates into results). It is a fascinating engine, but there is a ways to go yet.
With this sort of speedu-up due to cuDNN, at LTC, say 60'+ 15'', your LC0 should be about the level of Houdini 1.5a on 4 threads. It would be a fascinating match, H1.5 was a very tactically astute engine. In 10 games, there will be some LC0 wins, and I would be curious how LC0 missing mates in 3, and generally being very weak in tactics, especially in tactical puzzles, can beat H1.5, mostly positionally and on long term "plans". Seeing this cuDNN speed-up, I will get a GTX 1060 in less than 2 weeks.
You are greatly overestimating LC0 strength.
I run again 1000 games match between LC0 ID227 1000 playouts vs. SF depth 10, and LC0 didn't even reach 40%.
SF depth 10 is like 800-1000Elo weaker than H1.5 on 4 threads on 60'+15'' TC.
LC0 on cuDNN and 1060 is ~5000nps on 15x128 net, i.e. around 350'000 playouts per move on 60'+15'' TC.
If you really believe 8.5 doublings for LC0 is worth even close to 800 or 1000 Elo, what can I tell you, you are grossly mistaken. Just look at Andrey Chilantiev results in Tournaments section where it is clear than doubling in number of playouts for LC0 yields 50-60Elo.
You seem to extrapolate from very remote levels, which might vary greatly in Elo per doublings and many other things. I took 2900 CCRL level on close to 1060 card at 1'+1'', reported here by at least 2 posters. This was with non-CUDA 6x times slower engine. Now extrapolation begins: 50x time control --> 5.5 doublings. Scaling difference --> +35 Elo points per doubling (I got 60-65 at lower playouts), so at this LTC 60'+ 15'', the non-CUDA would perform at 3100 CCRL Elo level. With cuDNN, a factor of 6 in nps is 2.5 doublings, and I assumed at least 70 Elo points (accounting for better scaling too) --> additional 150 Elo points. 3100 + 150 = 3250 CCRL Elo level, similar to CCRL Elo level of Houdini 1.5a on 4 threads.

Well, when I get my GTX 1060, we might see who is closer, if no one will bother to test that in said conditions in say 10 games.

Milos
Posts: 3923
Joined: Wed Nov 25, 2009 12:47 am

Re: LCZero Accomplishments and Goals Thus Far

Post by Milos » Wed May 02, 2018 2:21 pm

Laskos wrote:With cuDNN engine, a factor of 6 is 2.5 doublings, and I assumed at least 70 Elo points (accounting for better scaling too) --> additional 150 Elo points. 3100 + 150 = 3250 CCRL Elo level, similar to CCRL Elo level of Houdini 1.5a on 4 threads.

Well, when I get my GTX 1060, we might see who is closer, if no one will bother to test that in said conditions in say 10 games.
We will probably need to wait some time for bug-fee cuDNN version. I now quickly tested tensorflow version too and it is affected the same as cuDNN version as also Albert Silver noted. Despite having 6x or more nps advantage cuDNN version loses in direct match with OpenCL version comfortably.
This is a total rewrite by mooskagh (Alexander Lyashuk) so probably SMP code that divides data in batches has some serious bugs. The code is not officially supported because Gian-Carlo is paranoid about using cuDNN with LC0 thinking it would break the licences which IMO is wrong, so he and than also Garry doesn't approve changing it to cuDNN.

stavros
Posts: 160
Joined: Tue Dec 02, 2014 12:29 am

Re: LCZero Accomplishments and Goals Thus Far

Post by stavros » Wed May 02, 2018 2:34 pm

CMCanavessi wrote:Here's my current list (232 gauntlet ongoing):

Code: Select all

   # PLAYER                                  :  RATING  PLAYED    W    D    L   (%)  D(%)  OppAvg  OppN  OppDiv
  93 Leela Chess Zero v0.7 ID 232 x64        :  2698.6     100   48   21   31    59    21  2635.6    25    25.0
  97 Leela Chess Zero v0.7 ID 195 x64        :  2666.9     200   86   45   69    54    23  2635.6    25    25.0
 101 Leela Chess Zero v0.7 ID 227 x64        :  2655.9     200   77   57   66    53    29  2635.6    25    25.0
 102 Leela Chess Zero v0.7 ID 219 x64        :  2654.0     200   74   62   64    53    31  2635.6    25    25.0
 108 Leela Chess Zero v0.7 ID 210 x64        :  2635.7     200   71   58   71    50    29  2635.6    25    25.0
 115 Leela Chess Zero v0.7 ID 189 x64        :  2604.4     200   61   61   78    46    31  2635.6    25    25.0
 119 Leela Chess Zero v0.7 ID 185 x64        :  2572.7     200   58   50   92    42    25  2635.6    25    25.0
 127 Leela Chess Zero v0.7 ID 176 x64        :  2521.6     400  188   80  132    57    20  2455.5    50    50.0
 132 Leela Chess Zero v0.7 ID 162 x64        :  2428.9     200  119   40   41    70    20  2275.4    25    25.0
 133 Leela Chess Zero v0.7 ID 169 x64        :  2420.2     200  118   38   44    69    19  2275.4    25    25.0
 136 Leela Chess Zero v0.7 ID 156 x64        :  2411.8     200  115   40   45    68    20  2275.4    25    25.0
 139 Leela Chess Zero v0.7 ID 150 x64        :  2367.1     200  104   40   56    62    20  2275.4    25    25.0
 147 Leela Chess Zero v0.5 ID 129 x64        :  2330.4     200  103   23   74    57    12  2275.4    25    25.0
 148 Leela Chess Zero v0.5 ID 124 x64        :  2328.5     200   95   38   67    57    19  2275.4    25    25.0
 149 Leela Chess Zero v0.6 ID 125 x64        :  2319.0     200   90   43   67    56    22  2275.4    25    25.0
 152 Leela Chess Zero v0.7 ID 145 x64        :  2300.2     200   87   39   74    53    20  2275.4    25    25.0
 162 Leela Chess Zero v0.6 ID 138 x64        :  2227.2     200   70   34   96    44    17  2275.4    25    25.0
 177 Leela Chess Zero v0.5 ID 123 x64        :  2177.7     400  215   58  127    61    15  2062.1    50    50.0
 202 Leela Chess Zero ID 122 x64             :  2050.6     200  136   24   40    74    12  1848.9    25    25.0
 227 Leela Chess Zero ID 103 x64             :  1950.3     200  113   25   62    63    13  1848.9    25    25.0
 232 Leela Chess Zero ID 116 x64             :  1921.8     200  107   23   70    59    12  1848.9    25    25.0
 235 Leela Chess Zero ID 98 x64              :  1900.0     200   99   28   73    57    14  1848.9    25    25.0
 251 Leela Chess Zero ID 80 x64              :  1806.3     200   76   26   98    45    13  1848.9    25    25.0
 265 Leela Chess Zero ID 69 x64              :  1739.7     200   62   21  117    36    11  1848.9    25    25.0
 283 Leela Chess Zero Gen 33 (ID 55) x64     :  1658.4     400  185   35  180    51     9  1632.6    45    43.5
 305 Leela Chess Zero Gen 30 (ID 44) x64     :  1495.8     200  106   22   72    59    11  1416.3    25    25.0
 313 Leela Chess Zero Gen 24 x64             :  1391.6     200   84   22   94    48    11  1416.3    25    25.0
 319 Leela Chess Zero Gen 20 x64             :  1308.5     200   64   27  109    39    14  1416.3    25    25.0
 331 Leela Chess Zero Gen 16 x64             :  1220.3     200   47   26  127    30    13  1416.3    25    25.0
 333 Leela Chess Zero Gen 14 x64             :  1176.5     200   42   20  138    26    10  1416.3    25    25.0
 337 Leela Chess Zero Gen 12 x64             :  1073.1     296   95   32  169    38    11  1174.8    49    46.3
 349 Leela Chess Zero Gen 10 x64             :   863.1      92   53   11   28    64    12   656.0    23    23.0
 352 Leela Chess Zero Gen 8 x64              :   794.5      92   45   17   30    58    18   656.0    23    23.0
 357 Leela Chess Zero Gen 6 x64              :   599.5      92   31   18   43    43    20   656.0    23    23.0
 361 Leela Chess Zero Gen 4 x64              :   368.8     150   43   18   89    35    12   621.3    15    15.0
hi. any link for that rating list?

User avatar
CMCanavessi
Posts: 907
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LCZero Accomplishments and Goals Thus Far

Post by CMCanavessi » Wed May 02, 2018 2:42 pm

stavros wrote:
CMCanavessi wrote:Here's my current list (232 gauntlet ongoing):

Code: Select all

   # PLAYER                                  :  RATING  PLAYED    W    D    L   (%)  D(%)  OppAvg  OppN  OppDiv
  93 Leela Chess Zero v0.7 ID 232 x64        :  2698.6     100   48   21   31    59    21  2635.6    25    25.0
  97 Leela Chess Zero v0.7 ID 195 x64        :  2666.9     200   86   45   69    54    23  2635.6    25    25.0
 101 Leela Chess Zero v0.7 ID 227 x64        :  2655.9     200   77   57   66    53    29  2635.6    25    25.0
 102 Leela Chess Zero v0.7 ID 219 x64        :  2654.0     200   74   62   64    53    31  2635.6    25    25.0
 108 Leela Chess Zero v0.7 ID 210 x64        :  2635.7     200   71   58   71    50    29  2635.6    25    25.0
 115 Leela Chess Zero v0.7 ID 189 x64        :  2604.4     200   61   61   78    46    31  2635.6    25    25.0
 119 Leela Chess Zero v0.7 ID 185 x64        :  2572.7     200   58   50   92    42    25  2635.6    25    25.0
 127 Leela Chess Zero v0.7 ID 176 x64        :  2521.6     400  188   80  132    57    20  2455.5    50    50.0
 132 Leela Chess Zero v0.7 ID 162 x64        :  2428.9     200  119   40   41    70    20  2275.4    25    25.0
 133 Leela Chess Zero v0.7 ID 169 x64        :  2420.2     200  118   38   44    69    19  2275.4    25    25.0
 136 Leela Chess Zero v0.7 ID 156 x64        :  2411.8     200  115   40   45    68    20  2275.4    25    25.0
 139 Leela Chess Zero v0.7 ID 150 x64        :  2367.1     200  104   40   56    62    20  2275.4    25    25.0
 147 Leela Chess Zero v0.5 ID 129 x64        :  2330.4     200  103   23   74    57    12  2275.4    25    25.0
 148 Leela Chess Zero v0.5 ID 124 x64        :  2328.5     200   95   38   67    57    19  2275.4    25    25.0
 149 Leela Chess Zero v0.6 ID 125 x64        :  2319.0     200   90   43   67    56    22  2275.4    25    25.0
 152 Leela Chess Zero v0.7 ID 145 x64        :  2300.2     200   87   39   74    53    20  2275.4    25    25.0
 162 Leela Chess Zero v0.6 ID 138 x64        :  2227.2     200   70   34   96    44    17  2275.4    25    25.0
 177 Leela Chess Zero v0.5 ID 123 x64        :  2177.7     400  215   58  127    61    15  2062.1    50    50.0
 202 Leela Chess Zero ID 122 x64             :  2050.6     200  136   24   40    74    12  1848.9    25    25.0
 227 Leela Chess Zero ID 103 x64             :  1950.3     200  113   25   62    63    13  1848.9    25    25.0
 232 Leela Chess Zero ID 116 x64             :  1921.8     200  107   23   70    59    12  1848.9    25    25.0
 235 Leela Chess Zero ID 98 x64              :  1900.0     200   99   28   73    57    14  1848.9    25    25.0
 251 Leela Chess Zero ID 80 x64              :  1806.3     200   76   26   98    45    13  1848.9    25    25.0
 265 Leela Chess Zero ID 69 x64              :  1739.7     200   62   21  117    36    11  1848.9    25    25.0
 283 Leela Chess Zero Gen 33 (ID 55) x64     :  1658.4     400  185   35  180    51     9  1632.6    45    43.5
 305 Leela Chess Zero Gen 30 (ID 44) x64     :  1495.8     200  106   22   72    59    11  1416.3    25    25.0
 313 Leela Chess Zero Gen 24 x64             :  1391.6     200   84   22   94    48    11  1416.3    25    25.0
 319 Leela Chess Zero Gen 20 x64             :  1308.5     200   64   27  109    39    14  1416.3    25    25.0
 331 Leela Chess Zero Gen 16 x64             :  1220.3     200   47   26  127    30    13  1416.3    25    25.0
 333 Leela Chess Zero Gen 14 x64             :  1176.5     200   42   20  138    26    10  1416.3    25    25.0
 337 Leela Chess Zero Gen 12 x64             :  1073.1     296   95   32  169    38    11  1174.8    49    46.3
 349 Leela Chess Zero Gen 10 x64             :   863.1      92   53   11   28    64    12   656.0    23    23.0
 352 Leela Chess Zero Gen 8 x64              :   794.5      92   45   17   30    58    18   656.0    23    23.0
 357 Leela Chess Zero Gen 6 x64              :   599.5      92   31   18   43    43    20   656.0    23    23.0
 361 Leela Chess Zero Gen 4 x64              :   368.8     150   43   18   89    35    12   621.3    15    15.0
hi. any link for that rating list?
Sure, here's the full rating list (constantly being updated though): https://pastebin.com/RJnsg9RM
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Daniel Shawul
Posts: 4055
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: LCZero Accomplishments and Goals Thus Far

Post by Daniel Shawul » Wed May 02, 2018 3:01 pm

Which branch has the cuDNN version? i see only tensorflow code for training.

Albert Silver
Posts: 2939
Joined: Wed Mar 08, 2006 8:57 pm
Location: Rio de Janeiro, Brazil

Re: LCZero Accomplishments and Goals Thus Far

Post by Albert Silver » Wed May 02, 2018 3:17 pm

Daniel Shawul wrote:Which branch has the cuDNN version? i see only tensorflow code for training.
https://github.com/mooskagh/leela-chess/tree/master/lc0
https://crem.xyz/lc0/
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."

brianr
Posts: 451
Joined: Thu Mar 09, 2006 2:01 pm

Re: LCZero Accomplishments and Goals Thus Far

Post by brianr » Wed May 02, 2018 3:20 pm

Daniel Shawul wrote:Which branch has the cuDNN version? i see only tensorflow code for training.
But the lc0 implementation (with cudnn) sounds like more of a rewrite, so it should not be used to measure the strength of play yet. The author is still trying to make it work compared to baseline lczero.

Daniel Shawul
Posts: 4055
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: LCZero Accomplishments and Goals Thus Far

Post by Daniel Shawul » Wed May 02, 2018 3:24 pm

Albert Silver wrote:
Daniel Shawul wrote:Which branch has the cuDNN version? i see only tensorflow code for training.
https://github.com/mooskagh/leela-chess/tree/master/lc0
https://crem.xyz/lc0/
Ok thanks!

User avatar
Laskos
Posts: 10839
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: LCZero Accomplishments and Goals Thus Far

Post by Laskos » Wed May 02, 2018 3:31 pm

Milos wrote:
Laskos wrote:With cuDNN engine, a factor of 6 is 2.5 doublings, and I assumed at least 70 Elo points (accounting for better scaling too) --> additional 150 Elo points. 3100 + 150 = 3250 CCRL Elo level, similar to CCRL Elo level of Houdini 1.5a on 4 threads.

Well, when I get my GTX 1060, we might see who is closer, if no one will bother to test that in said conditions in say 10 games.
We will probably need to wait some time for bug-fee cuDNN version. I now quickly tested tensorflow version too and it is affected the same as cuDNN version as also Albert Silver noted. Despite having 6x or more nps advantage cuDNN version loses in direct match with OpenCL version comfortably.
This is a total rewrite by mooskagh (Alexander Lyashuk) so probably SMP code that divides data in batches has some serious bugs. The code is not officially supported because Gian-Carlo is paranoid about using cuDNN with LC0 thinking it would break the licences which IMO is wrong, so he and than also Garry doesn't approve changing it to cuDNN.
Ah, ok, didn't know. I was really disquiet seeing those benches, and not having a reasonable GPU.

Dariusz Orzechowski
Posts: 44
Joined: Thu May 02, 2013 3:23 pm

Re: LCZero Accomplishments and Goals Thus Far

Post by Dariusz Orzechowski » Wed May 02, 2018 7:19 pm

I can confirm that:
- LCZ on GTX1060 and 1m+1s TC is now in CCRL 2900 ballpark (I tested ID228) when tested against engines in roughly 2900-3100 range,
- cuDNN version is much faster but doesn't look any stronger than OpenCL one.

I also tried a match of LCZ (the same HW and TC as above) against minimal SF9 (1 CPU, 32 MB hash, no book, no syzygy) with Noomen 2-moves openings and it was unfortunately almost absolute slaughter, around 96% for SF. I'm really curious to see how LCZ is going to overcome SF as many people seem certain it will happen. I'm not so sure that SF would go so quietly into the night.

Post Reply