LCZero: Progress and Scaling. Relation to CCRL Elo

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

nabildanial
Posts: 126
Joined: Thu Jun 05, 2014 5:29 am
Location: Malaysia

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by nabildanial »

JJJ wrote:I m not sure the FIDE estimate is accurate either, since no human could defeat recent version of LCzero, so probably none can defeat the ID 190 version.
The explanation on FIDE estimate in the Google Sheet makes perfect sense to me.

"The FIDE estimate indicates the strength of Leela playing at 1'+1'' time control, against a human that's playing at Classical time control. All estimates should be further adjusted based on the Elo scaling on the Scaling tab. For example, ID 125v7 (~2441 FIDE) would scale ~300 Elo with Rapid TC on this machine. Meaning at Rapid TC, it would play like a human player with a Classical Rating of ~2741 FIDE playing at Classical TC"
duncan
Posts: 12038
Joined: Mon Jul 07, 2008 10:50 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by duncan »

so elo 2552 on april 14th

now april 26 it is elo 2803. so a 251 elo gain in 12 days and 21 points a day gain. (including regression). To catch stockfish at 3563, another 760 points needed which should take about 5 weeks unless it slows down. so beginning of June. just maybe.
JJJ
Posts: 1346
Joined: Sat Apr 19, 2014 1:47 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by JJJ »

nabildanial wrote:
JJJ wrote:I m not sure the FIDE estimate is accurate either, since no human could defeat recent version of LCzero, so probably none can defeat the ID 190 version.
The explanation on FIDE estimate in the Google Sheet makes perfect sense to me.

"The FIDE estimate indicates the strength of Leela playing at 1'+1'' time control, against a human that's playing at Classical time control. All estimates should be further adjusted based on the Elo scaling on the Scaling tab. For example, ID 125v7 (~2441 FIDE) would scale ~300 Elo with Rapid TC on this machine. Meaning at Rapid TC, it would play like a human player with a Classical Rating of ~2741 FIDE playing at Classical TC"
So that's mean with just more time control than bullet, Leela is now stronger than Carslen playing with 90 min.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

One interesting aspect about the openings. I took Komodo 11.3.1 on 4 threads, which seems to perform the best in opening test suites, and maybe has the best opening phase of all standard engines, and pitted it against LC0 CPU ID201 on 4 threads (on a strong GPU it would be some 5 times faster) in three LTC games at 3600''+ 36''. In all three games the same happened: LC0 (as White) held its own until some tactics started to be involved, and not only that, it slowly gained some slight advantage in this initial quiet phase.

An example of an interrupted game, after LC0 started to lose clearly (all games were lost, obviously), Sicilian Steinitz variation:

[pgn][Event "My Tournament"]
[Site "?"]
[Date "2018.04.27"]
[Round "1"]
[White "LC0_CPU"]
[Black "Komodo 11.3.1"]
[Result "*"]
[ECO "B20"]
[Opening "Sicilian"]
[PlyCount "56"]
[Termination "unterminated"]
[TimeControl "3600+36"]
[Variation "Steinitz Variation"]

1. e4 {+0.13/23 63s} c5 {-0.15/29 184s} 2. g3 {+0.14/24 103s} Nc6 {0.00/29 106s}
3. Nf3 {+0.13/24 77s} Nf6 {-0.04/30 107s} 4. d3 {+0.05/24 91s}
d6 {-0.02/32 171s} 5. Bg2 {+0.14/24 103s} g6 {0.00/31 90s}
6. O-O {+0.11/25 107s} Bg7 {0.00/31 104s} 7. Re1 {+0.11/25 96s}
O-O {-0.04/31 133s} 8. c3 {+0.11/25 59s} Rb8 {0.00/29 175s} 9. a4 {+0.15/24 83s}
Be6 {0.00/30 105s} 10. d4 {+0.38/24 92s} cxd4 {-0.05/32 200s}
11. Nxd4 {+0.36/24 54s} Bd7 {0.00/28 100s} 12. Qe2 {+0.39/24 100s}
Qc8 {-0.02/30 217s} 13. Nxc6 {+0.52/24 74s} bxc6 {+0.12/30 109s}
14. a5 {+0.46/25 109s} Bg4 {+0.18/31 183s} 15. f3 {+0.52/23 67s}
Be6 {+0.11/31 118s} 16. a6 {+0.51/24 104s} Qc7 {+0.14/30 245s}
17. h3 {+0.68/24 101s} Nd7 {+0.32/30 208s} 18. Kh2 {+0.55/25 112s}
c5 {+0.42/31 199s} 19. f4 {+0.43/25 121s} c4 {+0.35/31 84s}
20. Be3 {+0.31/25 81s} Nc5 {+0.62/30 84s} 21. Na3 {+0.33/25 51s}
Nd3 {+1.32/30 104s} 22. Reb1 {+0.21/24 55s} Rb3 {+1.31/30 82s}
23. Nc2 {-0.46/25 123s} Rfb8 {+1.63/32 118s} 24. Nd4 {-0.20/24 62s}
Rxb2 {+1.61/30 78s} 25. Nxe6 {-0.19/24 54s} fxe6 {+1.54/31 105s}
26. Rxb2 {-0.14/24 1.0s} Rxb2 {+2.01/31 106s} 27. Qg4 {-0.14/24 1.0s}
Kf7 {+2.39/28 90s} 28. f5 {-0.52/24 72s} exf5 {+3.59/29 63s, No result} *[/pgn]

The thing is:
LC0 move 9.a2-a4

[D]1rbq1rk1/pp2ppbp/2np1np1/2p5/P3P3/2PP1NP1/1P3PBP/RNBQR1K1 b - - 0 9

Looking in databases of human games:

Chessbase Mega Database 2017:
White performance: 58.1% in 43 games

Chess Tempo Database:
White performance: 59.6% in 26 games

At this point is was Komodo exiting human theory with 9...Bc8-e6, not found in my large databases. Even SF9 sees some advantage for White (LC0) here, although standard engine analysis might be pretty worthless in this case.

In all three games, it was Komodo exiting the theory with some disadvantage, then, soon after, tactical complications completely demolished LC0.
Albert Silver
Posts: 3019
Joined: Wed Mar 08, 2006 9:57 pm
Location: Rio de Janeiro, Brazil

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Albert Silver »

It must really depend on the hardware. On my laptop, which according to the benchmark reaches 2031 NPS, it is about 2700+ CCRL. I just finished a 100-game match at g/10 against Nebula 2.0 (2726 CCRL) and Leela NN189 scored 50.5-49.5.

The laptop's hardware is i7-6700HQ (4 x 2.6GHz with HT) and GTX 980M (8GB version).
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
User avatar
George Tsavdaris
Posts: 1627
Joined: Thu Mar 09, 2006 12:35 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by George Tsavdaris »

Laskos wrote:One interesting aspect about the openings. I took Komodo 11.3.1 on 4 threads, which seems to perform the best in opening test suites, and maybe has the best opening phase of all standard engines, and pitted it against LC0 CPU ID201 on 4 threads (on a strong GPU it would be some 5 times faster) in three LTC games at 3600''+ 36''.
Better not to run LC0 with increment time controls as i think it doesn't support them. I think.
After his son's birth they've asked him:
"Is it a boy or girl?"
YES! He replied.....
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

George Tsavdaris wrote:
Laskos wrote:One interesting aspect about the openings. I took Komodo 11.3.1 on 4 threads, which seems to perform the best in opening test suites, and maybe has the best opening phase of all standard engines, and pitted it against LC0 CPU ID201 on 4 threads (on a strong GPU it would be some 5 times faster) in three LTC games at 3600''+ 36''.
Better not to run LC0 with increment time controls as i think it doesn't support them. I think.
It does support them.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Laskos »

The progress up to ID205:

Image
Werewolf
Posts: 1795
Joined: Thu Sep 18, 2008 10:24 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by Werewolf »

Very nice.
JJJ
Posts: 1346
Joined: Sat Apr 19, 2014 1:47 pm

Re: LCZero: Progress and Scaling. Relation to CCRL Elo

Post by JJJ »

It is normal LCzero outplay engines at the opening sometimes, and lack of tactics then. In the long run it will overcome the lack of tactics as well, it just needs many many many more games to try some sacrifice or some tactics with his actual level of play.