leela is official(?) better than sf9

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
Javier Ros
Posts: 186
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: leela is official(?) better than sf9

Post by Javier Ros » Mon Sep 10, 2018 5:42 pm

Laskos wrote: ↑
Sat Sep 08, 2018 9:45 am
stavros wrote: ↑
Sat Sep 08, 2018 9:29 am
 another shock! even with nvidia 980

https://groups.google.com/forum/?_escap ... K_oNeQHU2M
I am not sure, maybe I missed something, but I had 11199 tested. Ponder is ON there, which degrades SF8 to 2 cores, while not degrading Lc0. But the result is anyway strange.
Lc0 version 0.17 with net 11261 on Asus Geforce GTX 1060 3 Gb is slightly above Stockfish 8 with 6 threads for both engines on i7 4790S with 4 cores and 8 threads, ponder Off, Hash tables 1 Gb. for both. But on this hardware is inferior to Stockfish 9, I don´t know what happens on a 1080 and faster i7.

The key is the CUDA drivers for NVIDIA installed that multiply the performance by 5 or 6. Playing at 5 minutes plus 3 seconds 12-12. Playing at 25 minutes plus 10 sec lc0 13-Stockfish 8 11. Using the following set of 12 positions of AlphaZero



The games at 5 minutes plus 3 sec. are here

Stockfish_8 ½½½ 1½½½ 1½½½½½½½½00½ 10½½½ 12.0/24
Lc01711261 ½½½0½½½0½½½½½½½½ 1 1½0 1½½½ 12.0/24


Last edited by Javier Ros on Mon Sep 10, 2018 5:57 pm, edited 3 times in total.
The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.
Top
Javier Ros
Posts: 186
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: leela is official(?) better than sf9

Post by Javier Ros » Mon Sep 10, 2018 5:47 pm

And here the games at 25 minutes plus 10 seconds, same hardware

Lc01711261 ½½ 1½½½ 1½½½1½½01½½½½½½0½½ 13.0/24
Stockfish8 ½½0½½½0½½½0½½10½½½½½½1½½ 11.0/24

The first 12 games

The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.
Top
Javier Ros
Posts: 186
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: leela is official(?) better than sf9

Post by Javier Ros » Mon Sep 10, 2018 5:48 pm

And the last 12 games:

The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.
Top
Chessqueen
Posts: 1446
Joined: Wed Sep 05, 2018 12:16 am
Full name: George Pichard

Re: leela is official(?) better than sf9

Post by Chessqueen » Wed Sep 12, 2018 2:58 am

https://en.chessbase.com/post/gm-harry- ... ss-engines
Top
User avatar
Laskos
Posts: 10949
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: leela is official(?) better than sf9

Post by Laskos » Wed Sep 12, 2018 7:21 am

Javier Ros wrote: ↑
Mon Sep 10, 2018 5:42 pm
Laskos wrote: ↑
Sat Sep 08, 2018 9:45 am
stavros wrote: ↑
Sat Sep 08, 2018 9:29 am
 another shock! even with nvidia 980

https://groups.google.com/forum/?_escap ... K_oNeQHU2M
I am not sure, maybe I missed something, but I had 11199 tested. Ponder is ON there, which degrades SF8 to 2 cores, while not degrading Lc0. But the result is anyway strange.
Lc0 version 0.17 with net 11261 on Asus Geforce GTX 1060 3 Gb is slightly above Stockfish 8 with 6 threads for both engines on i7 4790S with 4 cores and 8 threads, ponder Off, Hash tables 1 Gb. for both. But on this hardware is inferior to Stockfish 9, I don´t know what happens on a 1080 and faster i7.

The key is the CUDA drivers for NVIDIA installed that multiply the performance by 5 or 6. Playing at 5 minutes plus 3 seconds 12-12. Playing at 25 minutes plus 10 sec lc0 13-Stockfish 8 11. Using the following set of 12 positions of AlphaZero

I seriously don't know what's the matter with these results. Reports of Lc0 on GTX 980 beating the hell out of SF8 on 4 i7 cores, your results, I am not getting them. Sure I use CUDA versions since months now. Let's check a bit. I used the same net ID11261 with Lc0 v17. What are Lc0 NPS on standard opening position after 300,000 nodes? On my GTX 1060 6GB I get about 3,700. The GPU card is not overclocked. Stockfish 8 on my four 3.8 GHz i7 Haswell cores (4 threads used) from the same initial position gets about 5.5 million NPS. Are those comparable to your numbers?

I use 3-mover pretty balanced starting positions extracted from GM and IM games. My result: Lc0 v17 11261 on GTX 1060 (using 2 threads) against SF8 on 4 i7 cores (4 threads):

Time control: 300'' + 3'' (identical to yours), ponder = off

Score of lc0_v17 11261 vs SF 8: +1 -7 =12 [0.350]
Elo difference: -107.54 +/- 96.06

20 of 20 games finished.

(I wanted to post the 20 games, but it seems they exceed the maximum numbers of characters allowed in a post.)

This result is consistent with my many earlier results showing that Lc0 with a good 11xxx net is fairly equal to Fire 7.1 on 4 cores, significantly below SF8 on 4 i7 cores. And Lc0 (with a good net) is a bit better than SF8 on 1 thread (in my conditions).
Were the tests showing stellar performance of Lc0 carried on laptops? Is it possible some severe CPU throttling to appear on these machines? My desktop is very stable, and I often check for temperatures, frequencies used and such, they seem all fine.
Top
User avatar
AdminX
Posts: 5610
Joined: Mon Mar 13, 2006 1:34 pm
Location: Acworth, GA
Contact:
Contact AdminX

Re: leela is official(?) better than sf9

Post by AdminX » Wed Sep 12, 2018 1:02 pm

OT: But here is a diiferent set of weights (datasets) to test with.

Introducing the CCRL Dataset

https://blog.lczero.org/2018/09/a-stand ... 2604790111
"Good decisions come from experience, and experience comes from bad decisions."
__________________________________________________________________
Ted Summers
Top
Javier Ros
Posts: 186
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: leela is official(?) better than sf9

Post by Javier Ros » Wed Sep 12, 2018 4:55 pm

Laskos wrote: ↑
Wed Sep 12, 2018 7:21 am
Javier Ros wrote: ↑
Mon Sep 10, 2018 5:42 pm
Laskos wrote: ↑
Sat Sep 08, 2018 9:45 am
stavros wrote: ↑
Sat Sep 08, 2018 9:29 am
 another shock! even with nvidia 980

https://groups.google.com/forum/?_escap ... K_oNeQHU2M
I am not sure, maybe I missed something, but I had 11199 tested. Ponder is ON there, which degrades SF8 to 2 cores, while not degrading Lc0. But the result is anyway strange.
Lc0 version 0.17 with net 11261 on Asus Geforce GTX 1060 3 Gb is slightly above Stockfish 8 with 6 threads for both engines on i7 4790S with 4 cores and 8 threads, ponder Off, Hash tables 1 Gb. for both. But on this hardware is inferior to Stockfish 9, I don´t know what happens on a 1080 and faster i7.

The key is the CUDA drivers for NVIDIA installed that multiply the performance by 5 or 6. Playing at 5 minutes plus 3 seconds 12-12. Playing at 25 minutes plus 10 sec lc0 13-Stockfish 8 11. Using the following set of 12 positions of AlphaZero

I seriously don't know what's the matter with these results. Reports of Lc0 on GTX 980 beating the hell out of SF8 on 4 i7 cores, your results, I am not getting them. Sure I use CUDA versions since months now. Let's check a bit. I used the same net ID11261 with Lc0 v17. What are Lc0 NPS on standard opening position after 300,000 nodes? On my GTX 1060 6GB I get about 3,700. The GPU card is not overclocked. Stockfish 8 on my four 3.8 GHz i7 Haswell cores (4 threads used) from the same initial position gets about 5.5 million NPS. Are those comparable to your numbers?

I use 3-mover pretty balanced starting positions extracted from GM and IM games. My result: Lc0 v17 11261 on GTX 1060 (using 2 threads) against SF8 on 4 i7 cores (4 threads):

Time control: 300'' + 3'' (identical to yours), ponder = off

Score of lc0_v17 11261 vs SF 8: +1 -7 =12 [0.350]
Elo difference: -107.54 +/- 96.06

20 of 20 games finished.
lc01711261.JPG
lc01711261.JPG (108.09 KiB) Viewed 2362 times
Yes, my numbers are very similar to yours. From the initial position lc0 gets 3322 NPS on my GTX 1060 3Gb. GPU card not overclocked, with default 2 threads, ponder Off, Hash tables 1 Gb
Stockfish 8 on 6 threads on i7 4790S (Desktop computer, well ventilated) 3.2GHz not overclocked (4 cores and 8 threads with Hyperthreading ON), ponder Off, Hash tables 1 Gb gets about 5.5 millions NPS.
Stockfish8.JPG
Stockfish8.JPG (69.93 KiB) Viewed 2361 times
Post the pgn with your positions and I will repeat the test on my computer and publish the games.

I have done another test with Stockfish 9, same conditions 5 min plus 3 secs against lc0 0.17 11261 with the following result

Lc0server11261Stockfish9_5mas3HT6CPU1024

Stockfish9 10½½ 110½½½ 1½½½ 1½½ 1½½½½½1 14.5/24
Lc01711261 01½½001½½½0½½½0½½0½½½½½0 9.5/24


First 12 games, next 12 in the following message.

Best regards

The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.
Top
Javier Ros
Posts: 186
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: leela is official(?) better than sf9

Post by Javier Ros » Wed Sep 12, 2018 4:57 pm

The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.
Top
User avatar
Laskos
Posts: 10949
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: leela is official(?) better than sf9

Post by Laskos » Wed Sep 12, 2018 6:02 pm

Javier Ros wrote: ↑
Wed Sep 12, 2018 4:55 pm
Laskos wrote: ↑
Wed Sep 12, 2018 7:21 am
Javier Ros wrote: ↑
Mon Sep 10, 2018 5:42 pm
Laskos wrote: ↑
Sat Sep 08, 2018 9:45 am
stavros wrote: ↑
Sat Sep 08, 2018 9:29 am
 another shock! even with nvidia 980

https://groups.google.com/forum/?_escap ... K_oNeQHU2M
I am not sure, maybe I missed something, but I had 11199 tested. Ponder is ON there, which degrades SF8 to 2 cores, while not degrading Lc0. But the result is anyway strange.
Lc0 version 0.17 with net 11261 on Asus Geforce GTX 1060 3 Gb is slightly above Stockfish 8 with 6 threads for both engines on i7 4790S with 4 cores and 8 threads, ponder Off, Hash tables 1 Gb. for both. But on this hardware is inferior to Stockfish 9, I don´t know what happens on a 1080 and faster i7.

The key is the CUDA drivers for NVIDIA installed that multiply the performance by 5 or 6. Playing at 5 minutes plus 3 seconds 12-12. Playing at 25 minutes plus 10 sec lc0 13-Stockfish 8 11. Using the following set of 12 positions of AlphaZero

I seriously don't know what's the matter with these results. Reports of Lc0 on GTX 980 beating the hell out of SF8 on 4 i7 cores, your results, I am not getting them. Sure I use CUDA versions since months now. Let's check a bit. I used the same net ID11261 with Lc0 v17. What are Lc0 NPS on standard opening position after 300,000 nodes? On my GTX 1060 6GB I get about 3,700. The GPU card is not overclocked. Stockfish 8 on my four 3.8 GHz i7 Haswell cores (4 threads used) from the same initial position gets about 5.5 million NPS. Are those comparable to your numbers?

I use 3-mover pretty balanced starting positions extracted from GM and IM games. My result: Lc0 v17 11261 on GTX 1060 (using 2 threads) against SF8 on 4 i7 cores (4 threads):

Time control: 300'' + 3'' (identical to yours), ponder = off

Score of lc0_v17 11261 vs SF 8: +1 -7 =12 [0.350]
Elo difference: -107.54 +/- 96.06

20 of 20 games finished.
lc01711261.JPG
Yes, my numbers are very similar to yours. From the initial position lc0 gets 3322 NPS on my GTX 1060 3Gb. GPU card not overclocked, with default 2 threads, ponder Off, Hash tables 1 Gb
Stockfish 8 on 6 threads on i7 4790S (Desktop computer, well ventilated) 3.2GHz not overclocked (4 cores and 8 threads with Hyperthreading ON), ponder Off, Hash tables 1 Gb gets about 5.5 millions NPS.
Stockfish8.JPG

Post the pgn with your positions and I will repeat the test on my computer and publish the games.

I have done another test with Stockfish 9, same conditions 5 min plus 3 secs against lc0 0.17 11261 with the following result

Lc0server11261Stockfish9_5mas3HT6CPU1024

Stockfish9 10½½ 110½½½ 1½½½ 1½½ 1½½½½½1 14.5/24
Lc01711261 01½½001½½½0½½½0½½0½½½½½0 9.5/24
I don't understand the discrepancy. Are you sure your SF9 uses 6 threads and not 1? This result would indicate 1 thread is used, but maybe I am doing something very wrong. OTOH, CCCC result, which I expected to come close to my result, because the CPU effective speed-up is about 8 compared to my 4 i7 cores, and GPU speed-up is again about 8 compared to to my GTX 1060, shows, like in my case, that Lc0 is the level of Fire 7.1. I use adjudication only for very long games, above 100 moves with evals in [-20cp, 20cp] range (usually very drawish long endgames). I use no win adjudication.

Here are the games, I managed to shorten the PGN in Scid:

Top
Javier Ros
Posts: 186
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: leela is official(?) better than sf9

Post by Javier Ros » Wed Sep 12, 2018 6:38 pm

Laskos wrote: ↑
Wed Sep 12, 2018 6:02 pm
I don't understand the discrepancy. Are you sure your SF9 uses 6 threads and not 1? This result would indicate 1 thread is used, but maybe I am doing something very wrong. OTOH, CCCC result, which I expected to come close to my result, because the CPU effective speed-up is about 8 compared to my 4 i7 cores, and GPU speed-up is again about 8 compared to to my GTX 1060, shows, like in my case, that Lc0 is the level of Fire 7.1. I use adjudication only for very long games, above 100 moves with evals in [-20cp, 20cp] range (usually very drawish long endgames). I use no win adjudication.
lc011120Stockfish9.JPG
lc011120Stockfish9.JPG (194.54 KiB) Viewed 2320 times
I am sure Stockfish uses 6 threads of the 8 threads (4 cores), you can see the 74% in the task manager for Stockfish 9 and 0% for lc0 due to Ponder Off and also the NPS counter is 8.7 million.

I use adjudication by Arena at -9.00

I have read in this link that the net 11120 is better than the rest of 11xxx and I am testing it right now. At this moment Stockfish 6.5-lc017 11120 6.5

https://groups.google.com/forum/#!topic ... IUjoNZAxVw

Version NN ELO Perf W D L Fire Komodo SF Score Games % Fire % Komodo % SF % Total
0.16 600 3428 56 186 118 61 48.5 39.5 149 360 50.83 40.42 32.92 41.39
0.16 643 3428 56 186 118 58.5 52 38.5 149 360 48.75 43.33 32.08 41.39
0.16 809 3428 56 188 116 56 51.5 42.5 150 360 46.67 42.92 35.42 41.67
0.16 695 3435 69 169 122 66 45.5 42 153.5 360 55 37.92 35 42.64
0.16 776 3435 61 187 112 59.5 54 41 154.5 360 49.58 45 34.17 42.92
0.16 928 3442 67 179 114 61.5 54.5 40.5 156.5 360 51.25 45.42 33.75 43.47
0.16 840 3456 64 198 98 67 53.5 42.5 163 360 55.83 44.58 35.42 45.28
0.17 1066 3463 77 184 99 70.5 51 47.5 169 360 58.75 42.5 39.58 46.94
0.17 1120 3470 77 189 94 68.5 54 49 171.5 360 57.08 45 40.83 47.64
0.17 1186 3449 68 183 109 59 57 43.5 159.5 360 49.17 47.5 36.25 44.31
The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.
Top
Post Reply

Return to “Computer Chess Club: General Topics”