Strong Leela nets competition

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

corres
Posts: 3657
Joined: Wed Nov 18, 2015 11:41 am
Location: hungary

Strong Leela nets competition

Post by corres »

I made a competition among strong Leela nets.
Parameters
Used Leela: v.0.22.0
GPUs: 2 x RTX 2060 OC
Threads: 4 X 4GHz
Backend: Multiplexing
BackendOptions: backend=cudnn-fp16,(gpu=0),(gpu=1)
NNCacheSize: 10000000
MaxPrefetchBatch: 64
Others are Default
Time Control: 1 min + 2 sec / move
50 opening positions played with alternated colors

Results
T40B.4-160 : 42872 = 4 : 3 (93 draw) 100 games
42872 : J13B.2-188 = 9 : 6 (85 draw) 100 games
T40B.4-160 : J13B.2-188 = 9 : 3 (88 draw) 100 games

In Points
T40B.4-160 net 13 points
42872 net 12 points
J13B.2-188 net 9 points

I made test with 384x30-T40-970 "big net" with disappointing result
J13B.2-188 : 384x30-T40-970 = 17(!) : 3 (80 draw) 100 games.
Seeing the result I did not make further test with it.
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Strong Leela nets competition

Post by Laskos »

corres wrote: Fri Nov 08, 2019 9:13 pm I made a competition among strong Leela nets.
Parameters
Used Leela: v.0.22.0
GPUs: 2 x RTX 2060 OC
Threads: 4 X 4GHz
Backend: Multiplexing
BackendOptions: backend=cudnn-fp16,(gpu=0),(gpu=1)
NNCacheSize: 10000000
MaxPrefetchBatch: 64
Others are Default
Time Control: 1 min + 2 sec / move
50 opening positions played with alternated colors

Results
T40B.4-160 : 42872 = 4 : 3 (93 draw) 100 games
42872 : J13B.2-188 = 9 : 6 (85 draw) 100 games
T40B.4-160 : J13B.2-188 = 9 : 3 (88 draw) 100 games

In Points
T40B.4-160 net 13 points
42872 net 12 points
J13B.2-188 net 9 points

I made test with 384x30-T40-970 "big net" with disappointing result
J13B.2-188 : 384x30-T40-970 = 17(!) : 3 (80 draw) 100 games.
Seeing the result I did not make further test with it.
J13B.4 is strong at LTC on strong hardware, or so it seems to me.
corres
Posts: 3657
Joined: Wed Nov 18, 2015 11:41 am
Location: hungary

Re: Strong Leela nets competition

Post by corres »

Laskos wrote: Fri Nov 08, 2019 9:21 pm ...
J13B.4 is strong at LTC on strong hardware, or so it seems to me.
Where we can see the results of J13b.4-xyz?
corres
Posts: 3657
Joined: Wed Nov 18, 2015 11:41 am
Location: hungary

Re: Strong Leela nets competition

Post by corres »

corres wrote: Fri Nov 08, 2019 9:13 pm I made a competition among strong Leela nets.
Parameters
Used Leela: v.0.22.0
GPUs: 2 x RTX 2060 OC
Threads: 4 X 4GHz
Backend: Multiplexing
BackendOptions: backend=cudnn-fp16,(gpu=0),(gpu=1)
NNCacheSize: 10000000
MaxPrefetchBatch: 64
Others are Default
Time Control: 1 min + 2 sec / move
50 opening positions played with alternated colors

Results
T40B.4-160 : 42872 = 4 : 3 (93 draw) 100 games
42872 : J13B.2-188 = 9 : 6 (85 draw) 100 games
T40B.4-160 : J13B.2-188 = 9 : 3 (88 draw) 100 games

In Points
T40B.4-160 net 13 points
42872 net 12 points
J13B.2-188 net 9 points

I made test with 384x30-T40-970 "big net" with disappointing result
J13B.2-188 : 384x30-T40-970 = 17(!) : 3 (80 draw) 100 games.
Seeing the result I did not make further test with it.
You can download the .zip file
wikisend.com
File ID: 621504
Password: leela
shrapnel
Posts: 1339
Joined: Fri Nov 02, 2012 9:43 am
Location: New Delhi, India

Re: Strong Leela nets competition

Post by shrapnel »

Playing big NNs against small NNs is one thing. Playing those NNs against Stockfish is a whole new Ball Game .
You "MAY" get different results if you play them against Stockfish.
"MAY" because your GPUs are weak and will be unable to fully utilize the powere of large NNs.
So, they may again lose on your Setup.
That's the Problem with testing of NNs.
One has to take into account BOTH the NN being used AND the GPU... TWO Factors involved.
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis
Damir
Posts: 2801
Joined: Mon Feb 11, 2008 3:53 pm
Location: Denmark
Full name: Damir Desevac

Re: Strong Leela nets competition

Post by Damir »

in TCEC Leela lost 2 games already…. it's play is crap, and seems no improvement has been made.. :) :) It is loser engine whose play only depends on strong GPU's... :) :)
corres
Posts: 3657
Joined: Wed Nov 18, 2015 11:41 am
Location: hungary

Re: Strong Leela nets competition

Post by corres »

shrapnel wrote: Sat Nov 09, 2019 12:15 pm Playing big NNs against small NNs is one thing. Playing those NNs against Stockfish is a whole new Ball Game .
You "MAY" get different results if you play them against Stockfish.
"MAY" because your GPUs are weak and will be unable to fully utilize the powere of large NNs.
So, they may again lose on your Setup.
That's the Problem with testing of NNs.
One has to take into account BOTH the NN being used AND the GPU... TWO Factors involved.
Sometimes you also should make some test, do you not think that?
I have no Twin RTX 2080 Ti only a twin RTX 2060 OC.
I put my openings on net and you should play two matches: a match (100 games) between the last Stockfish dev and your favorite big net as well as a match (100 games) between the last Stockfish dev and T40B.4-160 net with the same parameters. It takes about a day test, if you use 1 min + 2 sec / move TC (shorter time is bad!). If the big net will get better result against Stockfish than T40B.4-160 will, you are right.
It would be the exact evidence.
Any other is only ''bla-di-bla".
Nay Lin Tun
Posts: 708
Joined: Mon Jan 16, 2012 6:34 am

Re: Strong Leela nets competition

Post by Nay Lin Tun »

Damir wrote: Sat Nov 09, 2019 12:28 pm in TCEC Leela lost 2 games already…. it's play is crap, and seems no improvement has been made.. :) :) It is loser engine whose play only depends on strong GPU's... :) :)
I guess you are joking! Did you check what kind of custom positions they are testing for? Most of them are like that, I did not check all of them though!
Leela will probably play at 3000 rating! But not surprising!
NN are not trained for such custom positions that you wont see 1 in billion games.
[d]1nbqkbn1/1pppppp1/r6r/p6p/P6P/R6R/1PPPPPP1/1NBQKBN1 w - - 4 5
Damir
Posts: 2801
Joined: Mon Feb 11, 2008 3:53 pm
Location: Denmark
Full name: Damir Desevac

Re: Strong Leela nets competition

Post by Damir »

My response was to Shrapnel,,, since he keeps bashing all other engines and praise Leela... to show him that Lc0 is patzer engine too…. :) :)
User avatar
Nordlandia
Posts: 2821
Joined: Fri Sep 25, 2015 9:38 pm
Location: Sortland, Norway

Re: Strong Leela nets competition

Post by Nordlandia »

Try this position!

[d]3n1b2/p1n1k2p/1p6/5b2/7P/1P1PP1P1/P1P1K3/R3R3 w - - 0 1

Lyudmil Tsvetkov claimed black wins. But Lc0 did not manage to win against SF Dev and SF 10.

[pgn][Event "DESKTOP-N1IDKRV, Rapid 30m+30s"] [Site "Expert-Desktop"] [Date "2019.11.11"] [Round "1"] [White "Stockfish 10 64 BMI2"] [Black "Lc0 v0.22.0"] [Result "1/2-1/2"] [Annotator "0.00;-0.72"] [SetUp "1"] [FEN "3n1b2/p1n1k2p/1p6/5b2/7P/1P1PP1P1/P1P1K3/R3R3 w - - 0 1"] [PlyCount "142"] [TimeControl "1800+30"] 1. Rf1 {0.00/38 134 Both last book move} Bg4+ {-0.72/14 29} 2. Kd2 {-0.09/36 22 } Bg7 {-0.70/15 75 (Nf7)} 3. d4 {-0.10/36 77} Ne8 {-0.72/15 0 (Nf7)} 4. Rae1 { 0.09/38 18 (Kd3)} Bh5 {-0.76/15 75 (Nd6)} 5. c3 {0.00/35 39} Nf6 {-0.73/16 66} 6. e4 {-0.09/39 0} Bg6 {-0.73/15 3} 7. a4 {-0.16/40 313 (Kc1)} h5 {-0.65/13 118 (Ne6)} 8. Kc1 {-0.19/41 176} Kd7 {-0.84/17 22 (Bh6+)} 9. Re2 {0.00/45 43} Ng4 {-0.89/14 53} 10. Kd2 {0.00/46 0 (Kb2)} Nf7 {-1.16/15 184} 11. Kc2 { -0.19/48 0} Nd6 {-1.16/14 16} 12. Kb2 {-0.72/43 190} Ke7 {-1.18/16 1} 13. Rfe1 {-0.55/41 28} Kd8 {-1.18/15 23 (Bh6)} 14. Ka2 {-0.80/41 84} Bf8 {-1.18/17 1 (Bh6)} 15. Ka1 {-0.75/45 104 (Rf1)} Be7 {-1.20/12 245 (Nb7)} 16. Ka2 {-1.09/41 56} Kc8 {-1.18/12 110 (Bf8)} 17. Kb2 {-1.07/44 61} Kd8 {-1.17/12 59} 18. Ka1 { -1.05/42 0 (Ka2)} Kd7 {-1.17/11 76} 19. Kb2 {-1.06/42 0 (Ka2)} Ke8 {-1.16/11 61 (Kd8)} 20. Ka1 {-0.98/43 55} Bf8 {-1.16/12 74 (Nb7)} 21. Kb2 {-0.98/45 133 (Ka2)} Be7 {-1.16/12 32 (Kd7)} 22. Ka1 {-0.97/45 23} Kd8 {-1.15/12 84 (Bf8)} 23. Ka2 {-0.87/42 19} Kd7 {-1.15/12 77} 24. Ka1 {-0.83/46 0 (Kb2)} Kc8 { -1.16/11 63 (Bf8)} 25. Kb2 {-0.83/44 29} Kd7 {-1.15/12 36} 26. Ka1 {-0.77/45 0 (Ka2)} a6 {-1.14/12 84 (Bf8)} 27. Kb2 {-0.32/36 24 (Ka2)} Bf8 {-1.13/12 47 (Bd8)} 28. Rf1 {-1.19/41 458 (Ka1)} Bh6 {-1.12/13 49} 29. Rfe1 {-1.27/37 0} Kd8 {-1.11/12 63 (Kc6)} 30. Ka2 {-0.80/39 78} Kc8 {-1.10/13 57 (Kc7)} 31. Kb2 { -1.57/38 114} Bf8 {-1.10/14 48 (Kc7)} 32. Ka1 {-1.18/38 41 (Rf1)} Kd7 {-1.11/ 12 110 (Kc7)} 33. Ka2 {-1.39/40 68} Kc8 {-1.10/13 71 (Bh6)} 34. Kb2 {-1.73/41 99 (Rf1)} Kd8 {-1.10/12 73 (Kd7)} 35. Rf1 {-1.64/43 57 (Ka2)} Bh6 {-1.11/12 40 (Ke7)} 36. Rfe1 {-1.50/43 39} Ke8 {-1.09/12 71 (Kc7)} 37. Ka2 {-1.34/42 94} Kd8 {-1.08/13 66 (Kd7)} 38. Kb2 {-1.62/40 57} Kd7 {-1.08/13 75 (Kc7)} 39. Ka3 { -1.05/38 37 (Ka2)} Bf8 {-1.09/12 34} 40. Ka2 {-1.00/37 1 (Kb2)} Be7 {-1.08/12 92 (Ke7)} 41. Kb2 {-1.20/39 67} Kd8 {-1.09/13 21} 42. Ka2 {-0.94/38 2} Bf8 { -1.08/12 73 (Kc8)} 43. Kb2 {-1.20/40 42 (Ka1)} Ke8 {-1.09/13 37 (Kc7)} 44. Ka1 {-1.05/41 45 (Ka2)} Bh6 {-1.08/12 89} 45. Ka2 {-1.01/41 4} Kd7 {-1.06/13 60 (Ke7)} 46. Kb2 {-1.06/42 72 (Ka3)} Kd8 {-1.06/12 54 (Kc6)} 47. Ka3 {-0.88/35 18 (Ka2)} Bf8 {-1.06/12 72} 48. Kb2 {-0.85/43 0} Be7 {-1.07/12 49 (Ke8)} 49. Ka2 {-0.87/38 97} Ke8 {-1.08/16 0 (Kc7)} 50. Kb2 {-1.26/41 96} Bf8 {-1.08/14 34 (Bh7)} 51. Ka2 {-1.08/37 27} Kd7 {-1.08/14 84} 52. Kb2 {-0.82/42 0 (Ka1)} Bh6 {-1.08/12 48 (Bh7)} 53. Ka2 {-1.31/38 60} Kc8 {-1.06/13 59 (Kc7)} 54. Kb2 { -1.31/37 45} Kb7 {-1.06/13 56 (Kb8)} 55. Ka3 {-0.50/34 16 (Ka2)} Kc7 {-1.06/12 34} 56. Kb2 {-0.71/36 7 (Ka2)} Kc8 {-1.06/12 47 (b5)} 57. Ka2 {-0.90/37 35} Kc7 {-1.05/13 44 (Kb7)} 58. Kb2 {-0.73/40 41 (Ka3)} Kc8 {-1.05/12 42 (Bh7)} 59. Ka2 {-0.69/38 21} a5 {-1.05/12 14 (Kb7)} 60. Ka3 {0.00/46 18} Kd7 {-1.01/11 68 (Kc7)} 61. Kb2 {0.00/49 24 (Ka2)} Ke8 {-0.99/10 36 (Kc6)} 62. Ka1 {0.00/50 27 (Ka3)} Bf8 {-0.99/11 39 (Kd7)} 63. Ka2 {0.00/47 22} Kd8 {-0.98/11 31 (Ke7)} 64. Kb2 {0.00/48 27 (Ka1)} Bh6 {-0.95/11 35 (Ke7)} 65. Ka3 {-0.22/48 121 (Ka2)} Nc8 {-0.94/10 36 (Kc7)} 66. Ka2 {-0.47/39 66} Nd6 {-0.93/12 29} 67. Ka3 {-0.53/41 2 } Bh7 {-0.92/11 36 (Be8)} 68. Ka2 {0.00/45 22 (Kb2)} Kd7 {-0.90/11 30} 69. Ka3 {-0.39/38 35} Bf8 {-0.90/12 22 (Ke7)} 70. Kb2 {-0.77/37 56} Kd8 {-0.88/12 30 (Bg7)} 71. Ka1 {-0.59/34 17 (Ka2)} Ke8 {-0.87/11 36 (Bg6) adjud. Draw accepted} 1/2-1/2[/pgn]

Also he claimed that white wins here too.

[d]rk1r4/1b1p2pp/p1pPpp2/Pp2P2q/1P1P1P2/3BBN2/5RPP/R5K1 w - - 0 24

5+5 Blitz | Lc0 (2080 Super) vs 7-core SF Dev

[pgn][Event "?"] [Site "i7-5960X 4.5GHz"] [Date "2019.11.11"] [Round "?"] [White "lc0"] [Black "stockfish_x64_bmi2"] [Result "1-0"] [FEN "rk1r4/1b1p2pp/p1pPpp2/Pp2P2q/1P1P1P2/3BBN2/5RPP/R5K1 w - - 0 24"] [GameDuration "00:15:59"] [GameEndTime "2019-11-11T08:45:58.797 Vest-Europa (normaltid)"] [GameStartTime "2019-11-11T08:29:59.274 Vest-Europa (normaltid)"] [PlyCount "76"] [SetUp "1"] [Termination "adjudication"] [TimeControl "300+5"] 1. Re1 {+0.87/13 8.6s} Rf8 {+1.33/34 28s} 2. h3 {+0.97/13 16s} Re8 {+1.43/34 77s} 3. Bc2 {+1.26/11 13s} f5 {+0.78/35 31s} 4. Rc1 {+1.32/13 8.8s} Qf7 {+0.84/32 14s} 5. Ne1 {+1.56/12 14s} h6 {+0.84/30 6.0s} 6. Nd3 {+1.38/13 28s} Rg8 {+0.28/35 30s} 7. Nc5 {+1.83/10 9.0s} g5 {+1.22/27 6.3s} 8. Kh2 {+1.79/14 23s} g4 {0.00/37 16s} 9. hxg4 {+2.06/13 6.7s} Rxg4 {0.00/39 5.0s} 10. Bd1 {+2.08/14 8.0s} Rg8 {0.00/36 5.3s} 11. Rf3 {+2.13/12 13s} Qg6 {-0.49/40 42s} 12. Rc2 {+2.19/13 12s} Qh5+ {0.00/41 5.5s} 13. Kg1 {+2.18/13 8.2s} Rg7 {-1.39/43 52s} 14. Rd2 {+2.24/12 13s} Bc8 {-1.58/35 5.2s} 15. Bf2 {+2.29/12 12s} Qg4 {-1.38/30 3.5s} 16. Be1 {+2.33/12 12s} Ka7 {-2.78/36 31s} 17. Rg3 {+2.50/16 21s} Qxg3 {-2.54/36 4.0s} 18. Bxg3 {+2.46/17 8.4s} Rxg3 {-3.73/39 18s} 19. Bh5 {+2.51/17 4.6s} Rg7 {-4.15/33 9.5s} 20. Be8 {+2.57/17 10.0s} h5 {-4.39/32 5.2s} 21. Rd3 {+2.62/16 9.7s} h4 {-4.79/31 4.5s} 22. Rh3 {+2.65/15 11s} Rg4 {-4.81/30 3.6s} 23. g3 {+2.75/13 27s} Rg8 {-5.06/33 7.0s} 24. Bxd7 {+2.94/13 22s} Bxd7 {-5.10/30 3.3s} 25. Nxd7 {+2.95/14 6.1s} Rg6 {-6.84/31 6.7s} 26. Kf2 {+2.96/13 13s} Rd8 {-6.87/33 5.0s} 27. Nc5 {+3.01/13 13s} Rdg8 {-7.33/31 5.0s} 28. d7 {+3.07/13 7.1s} R6g7 {-6.07/28 2.5s} 29. d5 {+3.18/12 14s} cxd5 {-6.25/30 4.3s} 30. Rxh4 {+3.89/10 16s} Re7 {-5.85/31 4.2s} 31. Rh6 {+4.71/8 16s} Rd8 {-6.06/30 4.8s} 32. Ke3 {+4.71/9 15s} Ka8 {-8.37/30 9.1s} 33. Kd4 {+5.25/8 17s} Ka7 {-9.14/27 4.9s} 34. Nxe6 {+6.50/8 5.4s} Rdxd7 {-9.54/29 5.1s} 35. Nc5 {+7.25/8 4.9s} Rd8 {-9.91/27 5.0s} 36. Rxa6+ {+6.51/7 13s} Kb8 {-10.23/1 0s} 37. Rb6+ {+7.60/7 7.1s} Kc8 {-10.10/26 5.5s} 38. Rxb5 {+7.54/7 7.0s} Rh8 {-10.92/23 9.4s, White wins by adjudication} 1-0 [/pgn]