Lc0 wins, but Stockfish is still the best?

[mwyoung]

LC0 does not memorize positions.

But "memorizing" positions wouldn't be the justification for giving SF an opening book. The justification would be Lc0 uses 100MB+ more than SF, so give SF 100MB+ to use however SF people want.

Give SF 100 MB more hash then.
It seems to me that someone is angling for more advantage any way they can get it.

If you give SF a great book, and you don't give it to LC0, then who won the contest, SF or the book? (assuming it wins).

What needs to be tested here. Are some saying because Lc0 learns how to play for the Starting Position. This is somehow unfair to Stockfish? When Stockfish was turned to play with a good opening book.

And this point is not without some merit. As this has been the testing protocol. So CPU engines were never designed to play good with out a book. If it was the testing protocol to test with a short books or no books. Programmers would have spent more time turning for better for opening play.

We can test Stockfish with a good and big opening book. And have Lc0 play with no opening book. Or test both with allowing 20 moves of opening play. And see how much this changes the results.

To be fair it seems we should test this idea with both engine being allowed to play a 20 move opening book.

[Modern Times]

So CPU engines were never designed to play good with out a book.

Lc0 should learn to play chess960 !

[Javier Ros]

And remember, that Stockfish dev is 1167872 bytes! How much is current Lc0? 100 megabytes + cuda things?

This is a very important point.

You could argue SF should then get a 100MB opening book. If it did that it would not lose any games to current LC0.

This is not true.
I have made the following experiment. Using the pgn Balsa25Top, Lc0 without book and BrainFish with Cerebellum light Book with best move activated.
So that the pgn positions provide variety and the book of BrainFish avoid opening errors by Stockfish. From the pgn positions BrainFish played following Cerebellum book while Lc0 started thinking by itself.

At 5 minutes plus 3 seconds Lc0 on GTX1060x2 and Brainfish on i7 980X 6 cores

1 BrainFish_181219 +50 +14/=32/-4 60.00% 30.0/50
2 Lc0_41439x2 -50 +4/=32/-14 40.00% 20.0/50

https://sites.google.com/site/unoallavo ... ects=0&d=1

At 25 minutes plus 10 seconds GTX1060x2

1 BrainFish_181219 +27 +6/=42/-2 54.00% 27.0/50
2 Lc0_41439x2 -27 +2/=42/-6 46.00% 23.0/50

https://sites.google.com/site/unoallavo ... ects=0&d=1

At 5 minutes plus 3 seconds Lc0 on RTX2070 and Brainfish on i7 7490S 6 cores

1 BrainFish_181219_x64 +6 +6/=39/-5 51.00% 25.5/50
2 Lc0_41439 -6 +5/=39/-6 49.00% 24.5/50

https://sites.google.com/site/unoallavo ... ects=0&d=1

At 25 minutes plus 10 seconds RTX2070

1 Lc0_41439 +13 +4/=44/-2 52.00% 26.0/50
2 BrainFish_181219 -13 +2/=44/-4 48.00% 24.0/50

https://sites.google.com/site/unoallavo ... ects=0&d=1

In spite of the victory of Lc0 on RTX2070 at 25+10, BrainFish won the following game in only 22 movs.
Lc0 started to compute at move 6 while BrainFish begun at move 10.

[pgn][Event "CerebellumLc041439"] [Site "J-PC"] [Date "2019.03.10"] [Round "55"] [White "BrainFish_181219_x64_bmi2"] [Black "Lc0_41439"] [Result "1-0"] [ECO "B12"] [WhiteElo "2200"] [BlackElo "2200"] [PlyCount "44"] [EventDate "2019.??.??"] [TimeControl "1500+10"] 1. e4 c6 2. d4 d5 3. e5 Bf5 4. c3 e6 5. Nf3 Nd7 6. Be2 h6 {-0.55/13 43} 7. O-O a6 {-0.52/13 16} 8. h3 c5 {-0.52/13 63} 9. Be3 Bh7 {-0.50/17 38} 10. c4 { 0.79/30 44} Ne7 {-0.56/23 14} 11. Nc3 {0.94/30 27} cxd4 {-0.69/27 26} 12. Nxd4 {0.94/32 53} dxc4 {-0.69/18 0} 13. Bh5 {1.09/31 23} g6 {-0.93/20 43} 14. Bf3 { 1.34/30 34} Rb8 {-1.11/24 19} 15. Qa4 {1.33/32 21} g5 {-0.64/28 59} 16. Rfd1 { 1.64/30 19} Qc8 {0.24/20 66} 17. Rac1 {1.94/32 22} Ng6 {-0.11/27 51} 18. Ne4 { 5.30/34 19} b5 {-12.67/15 61} 19. Nxb5 {5.71/33 22} axb5 {-14.82/15 22} 20. Rxd7 {5.82/35 22} Bg7 {-14.76/14 28} 21. Qa7 {5.60/36 52} O-O {-14.99/13 24} 22. Nd6 {5.73/37 24} Nxe5 {normal -13.76/12 72 Black resigns} 1-0 [/pgn]

In the following, Lc0 started to compute at move 9 while BrainFish begun at move 19, despite which Lc0 won.


[pgn][Event "CerebellumLc041439"] [Site "J-PC"] [Date "2019.03.12"] [Round "66"] [White "Lc0_41439"] [Black "BrainFish_181219_x64_bmi2"] [Result "1-0"] [ECO "B90"] [WhiteElo "2200"] [BlackElo "2200"] [PlyCount "142"] [EventDate "2019.??.??"] [TimeControl "1500+10"] 1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. Be3 e5 7. Nb3 Be6 8. f3 Be7 9. Qd2 {0.42/14 24} Nbd7 10. g4 {0.42/16 34} O-O 11. O-O-O {0.40/20 44} b5 12. g5 {0.40/21 41} Nh5 13. Kb1 {0.40/19 0} Nb6 14. Na5 {0.42/22 34} Rc8 15. a3 {0.47/22 45} g6 16. Rg1 {0.30/21 75} Qc7 17. Nd5 {0.31/26 9} Nxd5 18. exd5 { 0.29/23 26} Bd7 19. c4 {0.47/21 44} bxc4 {0.00/35 76} 20. Rc1 {1.32/24 76} Bb5 {0.00/30 17} 21. Bxc4 {1.64/17 61} Bxc4 {-0.36/39 366} 22. Rxc4 {2.09/17 46} Qd7 {-0.40/32 21} 23. Nc6 {2.18/16 19} Rfe8 {-0.55/33 35} 24. Ka2 {1.96/13 133} Bf8 {0.00/30 24} 25. Qc2 {1.96/12 0} Ra8 {0.00/36 51} 26. h4 {1.90/14 35} Qh3 { 0.00/37 61} 27. Qg2 {2.36/14 55} Qd7 {-0.31/36 110} 28. Qg4 {2.30/14 99} Qc7 { 0.00/32 14} 29. a4 {2.33/16 19} f5 {0.00/38 92} 30. Qg2 {2.27/16 13} Qf7 { 0.00/35 97} 31. Rd1 {2.31/14 36} f4 {-0.10/37 32} 32. Bf2 {2.25/14 39} Ng7 { -0.43/38 79} 33. Qf1 {2.26/13 90} a5 {-0.18/37 55} 34. Qd3 {2.42/11 63} Nf5 { -0.67/36 151} 35. b3 {2.30/11 37} Be7 {-0.58/32 54} 36. Rdc1 {2.77/11 35} Bf8 { -0.73/33 85} 37. Rg1 {2.52/11 69} Ne7 {-0.81/36 53} 38. Rd1 {2.39/14 16} Nf5 { -0.67/33 13} 39. Ka3 {2.61/11 78} Kh8 {-0.52/28 51} 40. Bb6 {3.33/11 24} Nxh4 { -0.94/33 106} 41. Bxa5 {4.29/15 28} Be7 {-1.69/33 131} 42. Rg1 {5.04/14 31} Qf5 {-1.44/30 26} 43. Rc3 {5.03/16 30} h6 {-1.85/30 39} 44. gxh6 {5.00/14 0} Kh7 { -1.89/28 23} 45. Bb6 {5.07/15 64} Bf6 {-1.49/26 7} 46. Qe4 {5.05/16 8} Qh3 { -2.18/29 16} 47. Na7 {5.23/14 32} Reb8 {-2.25/27 43} 48. Bf2 {6.70/15 55} Nf5 { -2.73/31 23} 49. Rc7+ {7.11/16 24} Kxh6 {-3.32/30 12} 50. Rcc1 {6.69/16 12} Kg7 {-3.22/30 8} 51. Rg4 {6.72/14 0} Kh7 {-3.39/29 11} 52. Rcg1 {10.11/13 65} Rg8 { -4.03/27 10} 53. Nc6 {10.58/12 20} Rg7 {-4.53/29 10} 54. Qd3 {13.99/14 46} Qh2 {-4.86/27 9} 55. Qf1 {22.08/12 39} Kg8 {-4.77/29 7} 56. Rh1 {23.76/12 29} Ng3 { -6.62/29 12} 57. Rxg3 {23.61/14 15} fxg3 {-6.82/30 8} 58. Rxh2 {22.34/14 13} gxh2 {-6.76/30 11} 59. Bg3 {22.70/13 15} Rh7 {-7.30/31 10} 60. Bxh2 {22.78/12 14} Rxh2 {-5.82/26 6} 61. Qg1 {23.23/11 13} Rh6 {-7.40/28 12} 62. a5 {23.51/10 25} Kh7 {-7.19/24 10} 63. Ka4 {25.03/9 49} Rh4+ {-7.56/25 9} 64. b4 {23.87/10 22} Kh6 {-7.88/25 10} 65. Kb5 {26.17/8 13} Rf4 {-7.40/23 6} 66. a6 {24.15/9 16} Bh4 {-7.77/27 12} 67. Qh1 {23.66/9 15} Kg5 {-8.21/27 9} 68. a7 {26.02/8 13} Kf6 {-8.88/23 10} 69. Qh3 {28.04/8 8} Bf2 {-9.22/24 10} 70. Qd7 {27.92/9 8} Kg5 { -10.02/26 10} 71. Qxd6 {30.35/9 8} Rff8 {adjudication -10.39/25 9 Arena Adjudication} 1-0 [/pgn]

[Laskos]

LC0 does not memorize positions.

But "memorizing" positions wouldn't be the justification for giving SF an opening book. The justification would be Lc0 uses 100MB+ more than SF, so give SF 100MB+ to use however SF people want.

Give SF 100 MB more hash then.
It seems to me that someone is angling for more advantage any way they can get it.

If you give SF a great book, and you don't give it to LC0, then who won the contest, SF or the book? (assuming it wins).

What needs to be tested here. Are some saying because Lc0 learns how to play for the Starting Position. This is somehow unfair to Stockfish? When Stockfish was turned to play with a good opening book.

And this point is not without some merit. As this has been the testing protocol. So CPU engines were never designed to play good with out a book. If it was the testing protocol to test with a short books or no books. Programmers would have spent more time turning for better for opening play.

We can test Stockfish with a good and big opening book. And have Lc0 play with no opening book. Or test both with allowing 20 moves of opening play. And see how much this changes the results.

To be fair it seems we should test this idea with both engine being allowed to play a 20 move opening book.

Could you please try your very good quality RTX 2080ti in this benchmark I propose: Start Lc0 v21.1 then type the following:

setoption name Backend value cudnn-fp16
setoption name MinibatchSize value 512
setoption name WeightsFile value .\weights_run1_41687.pb.gz

pres enter, then type:

go

When you see that 10 million nodes were reached, copy the first output reported after 10 million nodes, to compare our NPS. My output was:

info depth 19 seldepth 57 time 232129 nodes 10456908 score cp 44 hashfull 246 nps 45047

Thus we can compare our GPUs, I sure cannot hope my numbers to even approach yours.

[jp]

This is not true.
I have made the following experiment. Using the pgn Balsa25Top, Lc0 without book and BrainFish with Cerebellum light Book with best move activated.
So that the pgn positions provide variety and the book of BrainFish avoid opening errors by Stockfish. From the pgn positions BrainFish played following Cerebellum book while Lc0 started thinking by itself.

What is the size of the Cerebellum light book?

In spite of the victory of Lc0 on RTX2070 at 25+10, BrainFish won the following game in only 22 movs.
Lc0 started to compute at move 6 while BrainFish begun at move 10.

Very similar to one of the TCEC games, I think. The wins are so opening-dependent.

[Javier Ros]

This is not true.
I have made the following experiment. Using the pgn Balsa25Top, Lc0 without book and BrainFish with Cerebellum light Book with best move activated.
So that the pgn positions provide variety and the book of BrainFish avoid opening errors by Stockfish. From the pgn positions BrainFish played following Cerebellum book while Lc0 started thinking by itself.

What is the size of the Cerebellum light book?

In spite of the victory of Lc0 on RTX2070 at 25+10, BrainFish won the following game in only 22 movs.
Lc0 started to compute at move 6 while BrainFish begun at move 10.

Very similar to one of the TCEC games, I think. The wins are so opening-dependent.

122 MB (128.798.256 bytes) is the size.

Yes, the wins are opening-dependent but due to the randomness of Lc0 it is very unlikely to get an opening book that avoid any defeat of Sf, and making that book would be a huge work.

[jp]

Okay, that's about the right size book, then.

We see in TCEC, etc. which openings SF doesn't do well in. You could craft a book to avoid those openings, but maybe that would be too "unfair" because that book is specific to Leela. A Leela book could do the same, of course.

Of course, it also matters whether you force a few plies or let them start from the initial position, when it'd be easier to avoid certain openings. When the reviewers made the DM guys play A0 against the crippled SF8 + book, it was mainly draws (IIRC > 90%).

mwyoung, the test you just started is going from the initial position, right?

[mwyoung]

Okay, that's about the right size book, then.

We see in TCEC, etc. which openings SF doesn't do well in. You could craft a book to avoid those openings, but maybe that would be too "unfair" because that book is specific to Leela. A Leela book could do the same, of course.

Of course, it also matters whether you force a few plies or let them start from the initial position, when it'd be easier to avoid certain openings. When the reviewers made the DM guys play A0 against the crippled SF8 + book, it was mainly draws (> 90%, I think).

mwyoung, the test you just started is going from the initial position, right?

Yes. I did not pick the opening, or know what is going to be played. Book is set to play the most solid mainline up to move 20 from GM play.

So far this theory seems busted. There is already enough games with the current score to say. Stockfish is not loosing because of the lack of a opening book. It is more likely that Lc0 is just better when playing Stockfish. Regardless of opening book length or size. This is what the data is showing.

[jp]

I posted in your other thread that a breakdown of score by openings at the end would be good.
Is the "Leela Ratio" about 1? (I ask for comparison of Lc with A0, not for comparison of Lc with SF.)

[lkaufman]

This gives Lc0 a huge hardware price advantage. On my testing system. I run a 16 core CPU for $799. At the same time Lc0 plays on a $1500 RTX 2080 ti. Is Lc0 better, yes but Lc0 needs every bit of the 2080 ti. And at almost twice the price to best Stockfish.

You can easily find an RTX 2080ti in $1200-$1250 range. Doesn't your CPU need a pretty large cooler to run consistently, without throttling on some very heavy and long CPU loads? That's another $70-$100 to almost $900 for CPU. GPUs don't need that. Also, RTX 2080ti doesn't seem to be the best deal around as GPU goes. I have my RTX 2070 for $550, 2.7 times cheaper than your expensive RTX 2080ti.

Let's compare apples to apples: Take the net 41687 (each net has its own speed curve, it is the latest I downloaded), put it on infinite from the standard opening position with "go". The engine should be Lc0 v0.21.1. Observe the speed after 10 million nodes. Mine is:

info depth 19 seldepth 57 time 232129 nodes 10456908 score cp 44 hashfull 246 nps 45047

I am curious what your 2.7 factor more expensive RTX 2080ti will show. If the scaling is good, two RTX 2070 would be a bit cheaper than one RTX 2080ti and I bet they would be faster.

As for CPU, I will buy this Summer or Autumn the third generation Ryzen 9 3850X, a 16 core CPU even stronger than your Threadripper 2950X, but for only $500.

All in all, Leela with latest T40 nets is very competitive with SF_dev strength-wise on fairly equal price-wise home-grade hardware.

I tried to replicate your test on my new laptop with RTX 2080. When I run it directly (without GUI) it doesn't use fp 16 and only gets about 12kNPS. So I run it with fp 16 in Fritz GUI on opening position, but still the NPS peaks at about 30knps and drops back to only 22knps by the time it has reached 10 million nodes. Since you got 45knps, shouldn't I get more than that on my 2080? Either I'm doing something wrong, or my GPU is not running full speed. I suppose that being a laptop might make it have to run slower, but I got the model that supposedly ran at the highest speed, not one that was slower but lighter. Any idea which is the explanation, or how I can find out what is wrong?