Are kiudee parameters the best?

[Ovyron]

Could you input UCI options and then "go" from the command prompt? Set also the desired weights file UCI option.

Here it is:

Code: Select all

       _
|   _ | |
|_ |_ |_| v0.23.2+git.c8d9095 built Dec 31 2019
setoption name WeightsFile value 256x20-mod-swa-1541000.pb
setoption name Threads value  4
setoption name CPuct value 2.147
setoption name CPuctBase value 18368.000000
setoption name CPuctFactor value 2.815
setoption name FpuValue value 0.443
setoption name PolicyTemperature value 1.607
go
Loading weights file from: 256x20-mod-swa-1541000.pb
Creating backend [blas]...
BLAS vendor: OpenBLAS.
OpenBLAS [NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Core2 MAX_THREADS=128].
OpenBLAS found 4 Core2 core(s).
OpenBLAS using 1 core(s) for this backend.
BLAS max batch size is 256.
info depth 1 seldepth 2 time 53796 nodes 3 score cp 32 hashfull 0 nps 1 tbhits 0
 pv e2e4 e7e5
info depth 2 seldepth 3 time 56480 nodes 5 score cp 29 hashfull 0 nps 0 tbhits 0
 pv e2e4 e7e5 g1f3
info depth 2 seldepth 4 time 60131 nodes 7 score cp 30 hashfull 0 nps 0 tbhits 0
 pv e2e4 e7e5 g1f3 b8c6
info depth 3 seldepth 5 time 61063 nodes 13 score cp 27 hashfull 0 nps 1 tbhits
0 pv e2e4 e7e5 b1c3 g8f6 g1f3
info depth 3 seldepth 6 time 65510 nodes 22 score cp 29 hashfull 0 nps 1 tbhits
0 pv e2e4 e7e5 b1c3 g8f6 g1f3 b8c6
info depth 3 seldepth 6 time 70571 nodes 32 score cp 28 hashfull 0 nps 1 tbhits
0 pv e2e4 e7e5 b1c3 b8c6 g1f3 g8f6 f3g5
info depth 4 seldepth 7 time 72264 nodes 43 score cp 28 hashfull 1 nps 2 tbhits
0 pv e2e4 e7e5 b1c3 g8f6 g1f3 b8c6 f1b5
info depth 4 seldepth 8 time 75813 nodes 71 score cp 26 hashfull 1 nps 2 tbhits
0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 c2c3 g8f6
info depth 4 seldepth 9 time 79224 nodes 92 score cp 28 hashfull 1 nps 3 tbhits
0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4
info depth 5 seldepth 9 time 79293 nodes 104 score cp 27 hashfull 1 nps 3 tbhits
 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 c2c3 g8f6 d2d3
info depth 4 seldepth 9 time 79740 nodes 119 score cp 27 hashfull 1 nps 4 tbhits
 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 c2c3 g8f6 d2d3
info depth 5 seldepth 10 time 82310 nodes 129 score cp 28 hashfull 1 nps 4 tbhit
s 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 c2c3 g8f6 d2d3 c5f2
info depth 5 seldepth 11 time 86490 nodes 183 score cp 28 hashfull 2 nps 5 tbhit
s 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6
info depth 6 seldepth 11 time 89219 nodes 237 score cp 28 hashfull 2 nps 6 tbhit
s 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 b1c3 f8d6 d2d3 e8g8 e1g1 f6g4
info depth 6 seldepth 12 time 91042 nodes 255 score cp 28 hashfull 2 nps 6 tbhit
s 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6 d7c6
info depth 6 seldepth 13 time 95767 nodes 333 score cp 29 hashfull 2 nps 7 tbhit
s 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 b1c3 f8d6 d2d3 e8g8 e1g1 a7a6 b5c6
info depth 7 seldepth 14 time 100683 nodes 445 score cp 29 hashfull 3 nps 9 tbhi
ts 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 b1c3 f8d6 d2d3 e8g8 e1g1 a7a6 b5c4
info depth 7 seldepth 15 time 105265 nodes 547 score cp 28 hashfull 3 nps 10 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 f1e1 d7d6 d2d3 h7h6 c2c3
info depth 7 seldepth 15 time 110323 nodes 592 score cp 28 hashfull 4 nps 10 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 f1e1 d7d6 d2d3 h7h6 c2c3
info depth 7 seldepth 16 time 111533 nodes 681 score cp 27 hashfull 4 nps 11 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 f1e1 d7d6 d2d3 e8g8 c2c3 a7a6
info depth 7 seldepth 17 time 116259 nodes 737 score cp 27 hashfull 4 nps 11 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 d2d3 d7d6 c2c3 a7a6 f1e1 h7h6 a
2a4 c5a7
info depth 8 seldepth 17 time 118840 nodes 817 score cp 27 hashfull 5 nps 12 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 d2d3 d7d6 c2c3 a7a6 f1e1 h7h6 a
2a4 c5a7
info depth 8 seldepth 18 time 119585 nodes 834 score cp 27 hashfull 5 nps 12 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 d2d3 d7d6 c2c3 a7a6 f1e1 h7h6 a
2a4 c5a7
info depth 8 seldepth 19 time 123510 nodes 873 score cp 27 hashfull 5 nps 12 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 d2d3 d7d6 c2c3 a7a6 f1e1 h7h6 a
2a4 c5a7
info depth 8 seldepth 20 time 127374 nodes 923 score cp 27 hashfull 5 nps 12 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1c4 f8c5 e1g1 g8f6 d2d3 d7d6 c2c3 a7a6 f1e1 h7h6 a
2a4 c5a7
info depth 8 seldepth 21 time 131098 nodes 971 score cp 27 hashfull 6 nps 12 tbh
its 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6 d7c6 d4e5 d6f5 d
1d8 e8d8 b1c3 f8e7 h2h3 h7h5 c1f4
info depth 8 seldepth 22 time 134272 nodes 1035 score cp 27 hashfull 6 nps 12 tb
hits 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6 d7c6 d4e5 d6f5
d1d8 e8d8 b1c3 f8e7 h2h3 h7h5 c1f4 f5h4
info depth 8 seldepth 23 time 138370 nodes 1091 score cp 27 hashfull 6 nps 12 tb
hits 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6 d7c6 d4e5 d6f5
d1d8 e8d8 b1c3 f8e7 h2h3 f5h4 f1d1 d8e8
info depth 9 seldepth 23 time 139187 nodes 1188 score cp 27 hashfull 6 nps 13 tb
hits 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6 d7c6 d4e5 d6f5
d1d8 e8d8 b1c3 f8e7 h2h3 h7h5 c1f4 d8e8 a1d1
info depth 9 seldepth 24 time 140547 nodes 1277 score cp 27 hashfull 7 nps 14 tb
hits 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6 d7c6 d4e5 d6f5
d1d8 e8d8 h2h3 f8e7 b1c3 h7h5 c1f4 d8e8 a1d1
stop
info depth 9 seldepth 24 time 142767 nodes 1311 score cp 27 hashfull 7 nps 14 tb
hits 0 pv e2e4 e7e5 g1f3 b8c6 f1b5 g8f6 e1g1 f6e4 d2d4 e4d6 b5c6 d7c6 d4e5 d6f5
d1d8 e8d8 b1c3 f8e7 h2h3 h7h5 c1f4 d8e8 a1d1 h5h4

Interestingly, under the Nibbler GUI it goes up to 23 nps very fast with default settings, though that's still x10 times slower than your phone, it's some progress!

[corres]

I made some tests with and without kiudee parameters.
Hardware
AMD Threadripper 16 x 4.0 GHz
2 x RTX 2060 OC (~1 x RTX 2080 Ti OC)
Common parameters of the tests
Chess GUI: DeepFritz 14
Hash (for Stockfish): 2048 MB
TC 1 min + 2 sec/move
Openings my TestBook50 with altered colors (2 x 50 = 100 games)
Tablebases: 6 men syzygy + 5 men Nalimov (only for GUI)

Parameters of Leela ver.0.22.0

"def" marked:
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
MaxPrefetch 64
Others are Leela Defaults

"ccc" marked (kiudee):
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
CPuct 2.147
CPuctBase 18368
CPuctFactor 2.815
FPUValue 0.443
MaxCollisionEvents 256
MaxPrefetch 64
PolicyTemperature 1.607
Others are Leela Defaults

Results 1.
ccc62078 : defJ13B.2-188 = 11 : 4 (85 draw) 100 games
def62078 : defJ13B.2-188 = 7 : 5 (88 draw) 100 games
(ccc gave +5 points)
Results 2.
ccc62078 : Stockfish191002 = 9 : 8 (83 draw) 100 games
def62078 : Stockfish191002 = 16 : 9 (75 draw) 100 games
(ccc gave -6 points)
Results 3.
ccc62078 : def62078 = 3 : 2 (95 draw) 100 games
(ccc gave +1 point)

From the above results it is obvious 62078net is stronger than J13B.2-188 and Stockfish191002 with and without kiudee parameters. I think the Elo of 62078net is valued down on the list of Leela nets.
On the other hand the effect of kiudee parameters is rather ambiguous so using it as a default settings is risky.
It is proposed to make tests with kiudee on the own system before using for analysis or engine-engine matches.
Note
You can download the test games from
wikisend.com
File ID 266998
Password leela

These are some newer results:
Results 4.
cccJ13B.2-188 : Stockfish191002 = 12 : 10 (78 draw) 100 games
defJ13B.2-188 : Stockfish191002 = 8 : 12 (80 draw) 100 games
(ccc gave -2 points)
Results 5.
cccT40B.4-160 : Stockfish191002 = 16 : 12 (72 draw) 100 games
defT40B.4-160 : Stockfish191002 = 10 : 10 (80 draw) 100 games
(ccc gave +4 points)
So the the ambiguous effect of kiudee parameters did not change.

[mwyoung]

I made some tests with and without kiudee parameters.
Hardware
AMD Threadripper 16 x 4.0 GHz
2 x RTX 2060 OC (~1 x RTX 2080 Ti OC)
Common parameters of the tests
Chess GUI: DeepFritz 14
Hash (for Stockfish): 2048 MB
TC 1 min + 2 sec/move
Openings my TestBook50 with altered colors (2 x 50 = 100 games)
Tablebases: 6 men syzygy + 5 men Nalimov (only for GUI)

Parameters of Leela ver.0.22.0

"def" marked:
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
MaxPrefetch 64
Others are Leela Defaults

"ccc" marked (kiudee):
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
CPuct 2.147
CPuctBase 18368
CPuctFactor 2.815
FPUValue 0.443
MaxCollisionEvents 256
MaxPrefetch 64
PolicyTemperature 1.607
Others are Leela Defaults

Results 1.
ccc62078 : defJ13B.2-188 = 11 : 4 (85 draw) 100 games
def62078 : defJ13B.2-188 = 7 : 5 (88 draw) 100 games
(ccc gave +5 points)
Results 2.
ccc62078 : Stockfish191002 = 9 : 8 (83 draw) 100 games
def62078 : Stockfish191002 = 16 : 9 (75 draw) 100 games
(ccc gave -6 points)
Results 3.
ccc62078 : def62078 = 3 : 2 (95 draw) 100 games
(ccc gave +1 point)

From the above results it is obvious 62078net is stronger than J13B.2-188 and Stockfish191002 with and without kiudee parameters. I think the Elo of 62078net is valued down on the list of Leela nets.
On the other hand the effect of kiudee parameters is rather ambiguous so using it as a default settings is risky.
It is proposed to make tests with kiudee on the own system before using for analysis or engine-engine matches.
Note
You can download the test games from
wikisend.com
File ID 266998
Password leela

These are some newer results:
Results 4.
cccJ13B.2-188 : Stockfish191002 = 12 : 10 (78 draw) 100 games
defJ13B.2-188 : Stockfish191002 = 8 : 12 (80 draw) 100 games
(ccc gave -2 points)
Results 5.
cccT40B.4-160 : Stockfish191002 = 16 : 12 (72 draw) 100 games
defT40B.4-160 : Stockfish191002 = 10 : 10 (80 draw) 100 games
(ccc gave +4 points)
So the the ambiguous effect of kiudee parameters did not change.

Are kuidee parameters the best? Well we already know the answer, if you understand how the parameters work. The answer is No! Because as there are many system types and time controls. There is no one best settings. But there are better settings then default. The main issue with the default settings is the too high value of the Policy Temperature setting. For most time controls, and most system. This setting is to high, unless you are doing hours long analysis. And in this case it is too low, along with Cpuct.

[corres]

I made some tests with and without kiudee parameters.
Hardware
AMD Threadripper 16 x 4.0 GHz
2 x RTX 2060 OC (~1 x RTX 2080 Ti OC)
Common parameters of the tests
Chess GUI: DeepFritz 14
Hash (for Stockfish): 2048 MB
TC 1 min + 2 sec/move
Openings my TestBook50 with altered colors (2 x 50 = 100 games)
Tablebases: 6 men syzygy + 5 men Nalimov (only for GUI)
Parameters of Leela ver.0.22.0
"def" marked:
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
MaxPrefetch 64
Others are Leela Defaults
"ccc" marked (kiudee):
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
CPuct 2.147
CPuctBase 18368
CPuctFactor 2.815
FPUValue 0.443
MaxCollisionEvents 256
MaxPrefetch 64
PolicyTemperature 1.607
Others are Leela Defaults
Results 1.
ccc62078 : defJ13B.2-188 = 11 : 4 (85 draw) 100 games
def62078 : defJ13B.2-188 = 7 : 5 (88 draw) 100 games
(ccc gave +5 points)
Results 2.
ccc62078 : Stockfish191002 = 9 : 8 (83 draw) 100 games
def62078 : Stockfish191002 = 16 : 9 (75 draw) 100 games
(ccc gave -6 points)
Results 3.
ccc62078 : def62078 = 3 : 2 (95 draw) 100 games
(ccc gave +1 point)
From the above results it is obvious 62078net is stronger than J13B.2-188 and Stockfish191002 with and without kiudee parameters. I think the Elo of 62078net is valued down on the list of Leela nets.
On the other hand the effect of kiudee parameters is rather ambiguous so using it as a default settings is risky.
It is proposed to make tests with kiudee on the own system before using for analysis or engine-engine matches.
Note
You can download the test games from
wikisend.com
File ID 266998
Password leela

These are some newer results:
Results 4.
cccJ13B.2-188 : Stockfish191002 = 12 : 10 (78 draw) 100 games
defJ13B.2-188 : Stockfish191002 = 8 : 12 (80 draw) 100 games
(ccc gave -2 points)
Results 5.
cccT40B.4-160 : Stockfish191002 = 16 : 12 (72 draw) 100 games
defT40B.4-160 : Stockfish191002 = 10 : 10 (80 draw) 100 games
(ccc gave +4 points)
So the the ambiguous effect of kiudee parameters did not change.

Are kuidee parameters the best? Well we already know the answer, if you understand how the parameters work. The answer is No! Because as there are many system types and time controls. There is no one best settings. But there are better settings then default. The main issue with the default settings is the too high value of the Policy Temperature setting. For most time controls, and most system. This setting is to high, unless you are doing hours long analysis. And in this case it is too low, along with Cpuct.

If you like to play the parameters of Leela do it.
But the large majority of Leela user do not like it, it is sure.
We only hope "kiudee" can find an universal and better setting for Leela. The universality is very important.

[mwyoung]

I made some tests with and without kiudee parameters.
Hardware
AMD Threadripper 16 x 4.0 GHz
2 x RTX 2060 OC (~1 x RTX 2080 Ti OC)
Common parameters of the tests
Chess GUI: DeepFritz 14
Hash (for Stockfish): 2048 MB
TC 1 min + 2 sec/move
Openings my TestBook50 with altered colors (2 x 50 = 100 games)
Tablebases: 6 men syzygy + 5 men Nalimov (only for GUI)
Parameters of Leela ver.0.22.0
"def" marked:
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
MaxPrefetch 64
Others are Leela Defaults
"ccc" marked (kiudee):
threads 4
backend multiplexing
backendoptions (backend=cudnn-fp16,gpu=1),(backend=cudnn-fp16,gpu=2)
NNcacheSize 10000000
CPuct 2.147
CPuctBase 18368
CPuctFactor 2.815
FPUValue 0.443
MaxCollisionEvents 256
MaxPrefetch 64
PolicyTemperature 1.607
Others are Leela Defaults
Results 1.
ccc62078 : defJ13B.2-188 = 11 : 4 (85 draw) 100 games
def62078 : defJ13B.2-188 = 7 : 5 (88 draw) 100 games
(ccc gave +5 points)
Results 2.
ccc62078 : Stockfish191002 = 9 : 8 (83 draw) 100 games
def62078 : Stockfish191002 = 16 : 9 (75 draw) 100 games
(ccc gave -6 points)
Results 3.
ccc62078 : def62078 = 3 : 2 (95 draw) 100 games
(ccc gave +1 point)
From the above results it is obvious 62078net is stronger than J13B.2-188 and Stockfish191002 with and without kiudee parameters. I think the Elo of 62078net is valued down on the list of Leela nets.
On the other hand the effect of kiudee parameters is rather ambiguous so using it as a default settings is risky.
It is proposed to make tests with kiudee on the own system before using for analysis or engine-engine matches.
Note
You can download the test games from
wikisend.com
File ID 266998
Password leela

These are some newer results:
Results 4.
cccJ13B.2-188 : Stockfish191002 = 12 : 10 (78 draw) 100 games
defJ13B.2-188 : Stockfish191002 = 8 : 12 (80 draw) 100 games
(ccc gave -2 points)
Results 5.
cccT40B.4-160 : Stockfish191002 = 16 : 12 (72 draw) 100 games
defT40B.4-160 : Stockfish191002 = 10 : 10 (80 draw) 100 games
(ccc gave +4 points)
So the the ambiguous effect of kiudee parameters did not change.

Are kuidee parameters the best? Well we already know the answer, if you understand how the parameters work. The answer is No! Because as there are many system types and time controls. There is no one best settings. But there are better settings then default. The main issue with the default settings is the too high value of the Policy Temperature setting. For most time controls, and most system. This setting is to high, unless you are doing hours long analysis. And in this case it is too low, along with Cpuct.

If you like to play the parameters of Leela do it.
But the large majority of Leela user do not like it, it is sure.
We only hope "kiudee" can find an universal and better setting for Leela. The universality is very important.

A better universal setting is already know. What do all do from fat fritz to Kiudee have in common. A much lower policy temperature. All other settings are just other ways of narrowing the search to give you better depth without too many tactical oversights.

[corres]

...
A better universal setting is already know. What do all do from fat fritz to Kiudee have in common. A much lower policy temperature. All other settings are just other ways of narrowing the search to give you better depth without too many tactical oversights.

The present kiudee parameters are not universal and it is not better than default RTX parameters - It is obvious from my results. Only in the case of J13B.2-188 net kiudee gives unambiguously better results than the default RTX.

[mwyoung]

...
A better universal setting is already know. What do all do from fat fritz to Kiudee have in common. A much lower policy temperature. All other settings are just other ways of narrowing the search to give you better depth without too many tactical oversights.

The present kiudee parameters are not universal and it is not better than default RTX parameters - It is obvious from my results. Only in the case of J13B.2-188 net kiudee gives unambiguously better results than the default RTX.

Exactly! There are no best setting for every system and NN. It depends on system speed and time controls and NN. We are not in disagreement.

[M ANSARI]

Wouldn't it be possible to have an automated system check that looks at your computer hardware and then follow a set table to automatically tune your Lc0 to the best possible settings? I mean it should be easy enough to poll the computer for number of available cores and frequency and then also check the memory and GPU. This could be a tuning app that runs once you install the Lc0 engine. It might not be a perfect tune, but for sure it would be better than the default.

[corres]

Wouldn't it be possible to have an automated system check that looks at your computer hardware and then follow a set table to automatically tune your Lc0 to the best possible settings? I mean it should be easy enough to poll the computer for number of available cores and frequency and then also check the memory and GPU. This could be a tuning app that runs once you install the Lc0 engine. It might not be a perfect tune, but for sure it would be better than the default.

In principle it is possible.
There is an issue namely the power of a chess engine is depend not only on the hardware and time control but the analysed position too. As tests also prove kiudee parameters are more effective in the case of unbalanced positions and openings, respectively. Moreover the longer TC lowers the effect of kiudee parameters so their practical usability is rather questionable.
Another problem is if everybody use his own parameters it will be non-comparable the results of tests.

[Hai]

Wouldn't it be possible to have an automated system check that looks at your computer hardware and then follow a set table to automatically tune your Lc0 to the best possible settings? I mean it should be easy enough to poll the computer for number of available cores and frequency and then also check the memory and GPU. This could be a tuning app that runs once you install the Lc0 engine. It might not be a perfect tune, but for sure it would be better than the default.

Yes it is possible.
Everybody would get a 50 elo stronger LC0 now.
Maybe even 100 elo stronger when using 2 gpus.