LCZero: Progress and Scaling. Relation to CCRL Elo

[George Tsavdaris]

Thanks for the update Kai!

On the one hand, it's quite possible that a fundamental change to its MCTS implementation will be required at some point if it wants to compete at the highest level, and the work Daniel Shawul has done with Scorpio could prove quite useful in that case (well, it's fantastic work in any case; it's just in this case that it would benefit LC0 ).

Doesn't LeelaZero uses the implementation of the Deepmind paper for AlphaZero? I didn't read that paper but i thought it contained full information about their implementation, sorry if i'm mistaken.

[Werewolf]

Large improvement from ID69 to ID83.

Now, in only 100 games gauntlets against Zurichess Bern (2232 Elo CCRL) and BikJump v2.01 (2098 Elo CCRL), it performs at about 2200 Elo level at 1s/move and at about 2300 Elo level at 10s/move. On a full 4 core i7 CPU.

On my own testing I've got slightly lower results than that. I've been matching it against old dedicated machines from the 1990s and giving LCZ ID 83 an Nvidia 1060 card to run on and >100,000 rollouts per move (Game in 30).

Although it is impressive how it outplays the weaker dedicated machines, especially with exchange sacs for positional compensation, it often gets hit by their tactics.

[Date "2018.04.04"]
[White "Master"]
[Black "LCZ ID 83 Nvidia 1060 100K"]
[Result "1/2-1/2"]

[pgn] 1. d4 Nf6 2. c4 e6 3. Nc3 d5 4. Bg5 Be7 5. e3 O-O 6. Nf3 h6 7. Bxf6 Bxf6 8. Rc1
c6 9. Bd3 a6 10. O-O b5 11. b3 Nd7 12. e4 dxc4 13. bxc4 e5 14. d5 Nc5 15. Bb1
b4 16. Na4 Nxa4 17. Qxa4 c5 18. Qc6 Bg4 19. Qxc5 Bxf3 20. gxf3 Be7 21. Qc6 Rc8
22. Qxa6 Bc5 23. Rcd1 Qg5+ 24. Kh1 Qh5 25. Rd3 Qh3 26. Rc1 Bxf2 27. Rb3 Rc5 28.
d6 Bd4 29. Rd3 Rfc8 30. Rcd1 R5c6 31. Qb7 Rxc4 32. d7 Rd8 33. Qb5 Rc3 34. Rxc3
bxc3 35. Qe2 Rxd7 36. Bc2 Qh5 37. a4 g6 38. a5 Rb7 39. Kg2 Rb2 40. Rb1 Ra2 41.
a6 Qh4 42. Rb7 Kg7 43. Rxf7+ Kh8 44. Rf8+ Kg7 45. Rf7+ Kh8 46. Rf8+ Kh7 47.
Rf7+ Kg8 48. Rb7 Kh8 49. Rf7 Kg8 50. Rb7 Kh8 51. Rf7 h5 52. Rf8+ Kg7 53. Rc8
Kh7 54. Rc6 Kh6 55. Rc8 Rxa6 56. Rxc3 Ra2 57. Rc6 Qf4 58. Kh3 Ra3 59. Bd3 Ra1
60. Qg2 Rg1 61. Rxg6+ Kh7 62. Rg7+ Kh8 63. Rg8+ Kh7 64. Rg7+ Kh6 65. Rg6+ Kh7
66. Rg7+ [/pgn]
1/2-1/2


LCZ just completely missed Rf7+!

[Werewolf]

Large improvement from ID69 to ID83.

Now, in only 100 games gauntlets against Zurichess Bern (2232 Elo CCRL) and BikJump v2.01 (2098 Elo CCRL), it performs at about 2200 Elo level at 1s/move and at about 2300 Elo level at 10s/move. On a full 4 core i7 CPU.

It improved significantly positionally from ID69 (in only 3 days).

Tactically it seems very weak.

And doesn't seem to improve at all. The estimated CCRL Elo level on this tactical suite is about that of Stockfish at depth=3, or maybe 1400 CCRL Elo points. Something has to be done with its MCTS search, maybe on the lines outlined by Daniel Shawl.

It's tactics are abysmal and I cannot see how it can get a high rating if this isn't fixed.

Consider this position:

[pgn] 1. e4 e5 2. Nf3 d6 3. Nc3 g6 4. Bc4 Bg4 [/pgn]

A 1980s dedicated chess computer gets this in about a minute. A 1990s dedicated or PC takes only 1 second to find 5.Ne5!

LCZ ID 83 was given around a million rollouts, but it couldn't find the move.

[Michel]

I think the expectation is that ultimately the policy component of the NN will learn to recognize such tactical patterns. Then they will be found by the MCTS.

Time will tell if this will really happen.

That doesn't mean it wouldn't be interesting to make versions of lczero equipped with different search algorithms in the meantime.

[Werewolf]

I think the expectation is that ultimately the policy component of the NN will learn to recognize such tactical patterns. Then they will be found by the MCTS.

That's interesting because when I hear the term "weights", what pops into my head something similar to material weights done by auto-tuning.

Evidently it's much more sophisticated than that.

[Jesse Gersenson]

I think the expectation is that ultimately the policy component of the NN will learn to recognize such tactical patterns. Then they will be found by the MCTS.

That's interesting because when I hear the term "weights", what pops into my head something similar to material weights done by auto-tuning.

Evidently it's much more sophisticated than that.

It works just like changing weights for piece values but the difference is the heuristic isn't piece value but is something slightly more abstract and (seemingly) arbitrary.

[duncan]

so is lczero 'meant ' to make stockfish obsolete by 2019,or is this hype ?

[Jhoravi]

Large improvement from ID69 to ID83.

Now, in only 100 games gauntlets against Zurichess Bern (2232 Elo CCRL) and BikJump v2.01 (2098 Elo CCRL), it performs at about 2200 Elo level at 1s/move and at about 2300 Elo level at 10s/move. On a full 4 core i7 CPU.

On my own testing I've got slightly lower results than that. I've been matching it against old dedicated machines from the 1990s and giving LCZ ID 83 an Nvidia 1060 card to run on and >100,000 rollouts per move (Game in 30).

Although it is impressive how it outplays the weaker dedicated machines, especially with exchange sacs for positional compensation, it often gets hit by their tactics.

[Date "2018.04.04"]
[White "Master"]
[Black "LCZ ID 83 Nvidia 1060 100K"]
[Result "1/2-1/2"]

[pgn] 1. d4 Nf6 2. c4 e6 3. Nc3 d5 4. Bg5 Be7 5. e3 O-O 6. Nf3 h6 7. Bxf6 Bxf6 8. Rc1
c6 9. Bd3 a6 10. O-O b5 11. b3 Nd7 12. e4 dxc4 13. bxc4 e5 14. d5 Nc5 15. Bb1
b4 16. Na4 Nxa4 17. Qxa4 c5 18. Qc6 Bg4 19. Qxc5 Bxf3 20. gxf3 Be7 21. Qc6 Rc8
22. Qxa6 Bc5 23. Rcd1 Qg5+ 24. Kh1 Qh5 25. Rd3 Qh3 26. Rc1 Bxf2 27. Rb3 Rc5 28.
d6 Bd4 29. Rd3 Rfc8 30. Rcd1 R5c6 31. Qb7 Rxc4 32. d7 Rd8 33. Qb5 Rc3 34. Rxc3
bxc3 35. Qe2 Rxd7 36. Bc2 Qh5 37. a4 g6 38. a5 Rb7 39. Kg2 Rb2 40. Rb1 Ra2 41.
a6 Qh4 42. Rb7 Kg7 43. Rxf7+ Kh8 44. Rf8+ Kg7 45. Rf7+ Kh8 46. Rf8+ Kh7 47.
Rf7+ Kg8 48. Rb7 Kh8 49. Rf7 Kg8 50. Rb7 Kh8 51. Rf7 h5 52. Rf8+ Kg7 53. Rc8
Kh7 54. Rc6 Kh6 55. Rc8 Rxa6 56. Rxc3 Ra2 57. Rc6 Qf4 58. Kh3 Ra3 59. Bd3 Ra1
60. Qg2 Rg1 61. Rxg6+ Kh7 62. Rg7+ Kh8 63. Rg8+ Kh7 64. Rg7+ Kh6 65. Rg6+ Kh7
66. Rg7+ [/pgn]
1/2-1/2


LCZ just completely missed Rf7+!

Maybe LCZ actually expected Rf7+ with the intent of accepting the draw. What LCZ really missed was 42..Rxa6 instead of 42..Kg7. if White replies 43 QxR then white is mated starting with 43..Qf2+
I'm not sure if I miss something because I'm not using an engine.

[Werewolf]

Large improvement from ID69 to ID83.

Now, in only 100 games gauntlets against Zurichess Bern (2232 Elo CCRL) and BikJump v2.01 (2098 Elo CCRL), it performs at about 2200 Elo level at 1s/move and at about 2300 Elo level at 10s/move. On a full 4 core i7 CPU.

On my own testing I've got slightly lower results than that. I've been matching it against old dedicated machines from the 1990s and giving LCZ ID 83 an Nvidia 1060 card to run on and >100,000 rollouts per move (Game in 30).

Although it is impressive how it outplays the weaker dedicated machines, especially with exchange sacs for positional compensation, it often gets hit by their tactics.

[Date "2018.04.04"]
[White "Master"]
[Black "LCZ ID 83 Nvidia 1060 100K"]
[Result "1/2-1/2"]

[pgn] 1. d4 Nf6 2. c4 e6 3. Nc3 d5 4. Bg5 Be7 5. e3 O-O 6. Nf3 h6 7. Bxf6 Bxf6 8. Rc1
c6 9. Bd3 a6 10. O-O b5 11. b3 Nd7 12. e4 dxc4 13. bxc4 e5 14. d5 Nc5 15. Bb1
b4 16. Na4 Nxa4 17. Qxa4 c5 18. Qc6 Bg4 19. Qxc5 Bxf3 20. gxf3 Be7 21. Qc6 Rc8
22. Qxa6 Bc5 23. Rcd1 Qg5+ 24. Kh1 Qh5 25. Rd3 Qh3 26. Rc1 Bxf2 27. Rb3 Rc5 28.
d6 Bd4 29. Rd3 Rfc8 30. Rcd1 R5c6 31. Qb7 Rxc4 32. d7 Rd8 33. Qb5 Rc3 34. Rxc3
bxc3 35. Qe2 Rxd7 36. Bc2 Qh5 37. a4 g6 38. a5 Rb7 39. Kg2 Rb2 40. Rb1 Ra2 41.
a6 Qh4 42. Rb7 Kg7 43. Rxf7+ Kh8 44. Rf8+ Kg7 45. Rf7+ Kh8 46. Rf8+ Kh7 47.
Rf7+ Kg8 48. Rb7 Kh8 49. Rf7 Kg8 50. Rb7 Kh8 51. Rf7 h5 52. Rf8+ Kg7 53. Rc8
Kh7 54. Rc6 Kh6 55. Rc8 Rxa6 56. Rxc3 Ra2 57. Rc6 Qf4 58. Kh3 Ra3 59. Bd3 Ra1
60. Qg2 Rg1 61. Rxg6+ Kh7 62. Rg7+ Kh8 63. Rg8+ Kh7 64. Rg7+ Kh6 65. Rg6+ Kh7
66. Rg7+ [/pgn]
1/2-1/2


LCZ just completely missed Rf7+!

Maybe LCZ actually expected Rf7+ with the intent of accepting the draw. What LCZ really missed was 42..Rxa6 instead of 42..Kg7. if White replies 43 QxR then white is mated starting with 43..Qf2+
I'm not sure if I miss something because I'm not using an engine.

I was watching the analysis window carefully. It just totally missed Rf7+.
It considered 42...Ra6 but didn't play it because it thought it could do it next go with a better king position...failing to see the check on f7. After 42...Kg7 its eval was very high, then after the check it dropped, but even then didn't see a draw.

It did see the tactics on f2 though. Odd, isn't it?

[Dann Corbit]

so is lczero 'meant ' to make stockfish obsolete by 2019,or is this hype ?

Not the purpose of lczero.
It is a completely different way for a computer to play chess.
It uses completely different hardware.
Even if it should become stronger, it will only be useful for machines with the appropriate hardware.

At some point, the power of tensor arrays will also apply to ordinary computer chips as well, once some smart people figure out how to use them.