Page 1 of 6
A match between SF12+NNUE and Leele ver.0.26.2
Posted: Tue Sep 08, 2020 10:06 pm
by corres
My system is:
OS Windows 10 x64 bits
Processor Ryzen 9 3950x 16 x 4.0 GHz fixed, no power-boost, no SMT
Ram 32 GB
TRIAD = 1 x RTX 2080 Ti OC + 2 x RTX 2060 OC GPUs (chess power ~ 2 x RTX 2080 Ti)
Chess GUI Fritz 14
Other params:
SF12 +NNUE exe is optimized for avx2
Hash for CPU 2 GB
Net for NNUE nn-82215d0fd0df.nnue (the default)
Leela version is for CUDA 11 and cudnn 7.6.4
Leela parameters were "Kiudee" (Default) and Threads = 6 x 4.0 GHz, NNCachesize = 20000000,
Backend = Multiplexing
Net for Leela SV-384x30-T60-3010
Endgame databases 6 men syzygy and 5 men Nalimov for chess GUI
Opening Book My Mini-book with 100 opening position with alternated color.
Tournament time 1 min + 2 sec / move
Result is
SF12+NNUE : kTRIAD = 9 : 6 (185 draw) 200 games
Note
Leela speed with CUDA11 and cudnn 7.6.4 is faster about 30 % as her speed with CUDA 10 and cudnn 7.4.2
(the default one) on SV-384x30-T60-3010 net
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Tue Sep 08, 2020 11:53 pm
by mwyoung
corres wrote: ↑Tue Sep 08, 2020 10:06 pm
My system is:
OS Windows 10 x64 bits
Processor Ryzen 9 3950x 16 x 4.0 GHz fixed, no power-boost, no SMT
Ram 32 GB
TRIAD = 1 x RTX 2080 Ti OC + 2 x RTX 2060 OC GPUs (chess power ~ 2 x RTX 2080 Ti)
Chess GUI Fritz 14
Other params:
SF12 +NNUE exe is optimized for avx2
Hash for CPU 2 GB
Net for NNUE nn-82215d0fd0df.nnue (the default)
Leela version is for CUDA 11 and cudnn 7.6.4
Leela parameters were "Kiudee" (Default) and Threads = 6 x 4.0 GHz, NNCachesize = 20000000,
Backend = Multiplexing
Net for Leela SV-384x30-T60-3010
Endgame databases 6 men syzygy and 5 men Nalimov for chess GUI
Opening Book My Mini-book with 100 opening position with alternated color.
Tournament time 1 min + 2 sec / move
Result is
SF12+NNUE : kTRIAD = 9 : 6 (185 draw) 200 games
Note
Leela speed with CUDA11 and cudnn 7.6.4 is faster about 30 % as her speed with CUDA 10 and cudnn 7.4.2
(the default one) on SV-384x30-T60-3010 net
Very very interesting...
What out for the SF fanboys. They are vicious .
Code: Select all
Wins Losses Draws
9
6
185
Points Games
101.5
/
200
Winning percentage
50.75
Elo difference :
+5
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 12:04 am
by corres
Maybe the developers of Leela should modify the the used CUDA and cudnn .dll.
+30 % speed gives about 10-20 Elo.
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 12:13 am
by mwyoung
corres wrote: ↑Wed Sep 09, 2020 12:04 am
Maybe the developers of Leela should modify the the used CUDA and cudnn .dll.
+30 % speed gives about 10-20 Elo.
I am unclear on the Elo gain here. My 200 game test with SF12+NNUE and Leele ver.0.26.2.
SF 12 won by +24 Elo using only 1 RTX 2080ti vs 32 thread SF 12 on a 2950x.
But I was using NN J92-115 for Lc0.
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 12:28 am
by Scipion
Could you please post a SF12 lost game ?
Never seen one yet in all my engines tournaments.
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 12:29 am
by OliverBr
corres wrote: ↑Tue Sep 08, 2020 10:06 pm
Result is
SF12+NNUE : kTRIAD = 9 : 6 (185 draw) 200 games
This is no result at all, you need at least 5000 games in order to announce a result.
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 12:31 am
by mwyoung
OliverBr wrote: ↑Wed Sep 09, 2020 12:29 am
corres wrote: ↑Tue Sep 08, 2020 10:06 pm
Result is
SF12+NNUE : kTRIAD = 9 : 6 (185 draw) 200 games
This is no result at all, you need at least 5000 games in order to announce a result.
True unless the games are played by TCEC.
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 12:36 am
by corres
OliverBr wrote: ↑Wed Sep 09, 2020 12:29 am
corres wrote: ↑Tue Sep 08, 2020 10:06 pm
Result is
SF12+NNUE : kTRIAD = 9 : 6 (185 draw) 200 games
This is no result at all, you need at least 5000 games in order to announce a result.
Sorry, Oliver, I am not an official tester, I make tests for my interest only.
I report the parameters of my test, and others can level my test as they want.
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 12:52 am
by mwyoung
corres wrote: ↑Wed Sep 09, 2020 12:36 am
OliverBr wrote: ↑Wed Sep 09, 2020 12:29 am
corres wrote: ↑Tue Sep 08, 2020 10:06 pm
Result is
SF12+NNUE : kTRIAD = 9 : 6 (185 draw) 200 games
This is no result at all, you need at least 5000 games in order to announce a result.
Sorry, Oliver, I am not an official tester, I make tests for my interest only.
I report the parameters of my test, and others can level my test as they want.
A tester does not need 5000 games, unless the goal is only to maximize the Elo precision. A useful tool for engine developers.
What are you trying to answer.
1. What is the exact elo deference between A vs B.
2. Who is better only between A vs B, and this usually takes much fewer games to prove statistically.
Re: A match between SF12+NNUE and Leele ver.0.26.2
Posted: Wed Sep 09, 2020 1:01 am
by OliverBr
mwyoung wrote: ↑Wed Sep 09, 2020 12:52 am
What are you trying to answer.
1. What is the exact elo deference between A vs B.
2. Who is better only between A vs B, and this usually takes much fewer games to prove statistically.
Still you need a couple of thousand games.
Believe me, I can do them with a 32-core computer and no, some hundred games are not enough.