Page 1 of 6
Lc0: Kiudee setting is very strong
Posted: Mon Jan 20, 2020 2:38 pm
by pohl4711
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (
https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Re: Lc0: Kiudee setting is very strong
Posted: Mon Jan 20, 2020 3:51 pm
by corres
pohl4711 wrote: ↑Mon Jan 20, 2020 2:38 pm
...
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo.
But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic.
...
Is there "Armageddon Elo" and a method to calculate "Armageddon Elo"?
I do not know about that.
Note
Kiudee-settings may be good for "Armageddon" but what is the situation in the case of normal games?
Re: Lc0: Kiudee setting is very strong
Posted: Mon Jan 20, 2020 4:14 pm
by pohl4711
Armageddon means, all draws are counted as a win for Black. And the opening-positions give White an advantage (SALC: White can castle short, Black can castle long, only). Thats it. Because, there are no draws, the results are spreaded (more away from 50%-50%).
Here some tests from discord, with "normal" openings:
Code: Select all
tc=1s+0.1s, RTX 2070
"bonus settings" cpuct=2.147, fpu=0.443, pst=1.607, cpuct-base=18368, cpuct-factor=2.815
# PLAYER : RATING ERROR POINTS PLAYED (%) CFS(%) W D L
1 lc0.net.58613.kiudee_bonus : 86 10 836.5 1422 58.8 100 459 755 208
2 lc0.net.58613.default : 33 9 699.0 1422 49.2 55 320 758 344
3 lc0.net.LD2.kiudee_bonus : 32 10 696.5 1422 49.0 100 308 777 337
4 lc0.net.LD2.default : 0 9 612.0 1422 43.0 --- 240 744 438
(both Nets are small (10x128). +32 and +53 Elo gain = +42 Elo average gain)
Gauntlet: J13B.2-188 vs lc0.net.62013 (default) / lc0.net.62013-tuned @kiudee
LC0-version: lc0-v0.23.2, Backend=cudnn-fp16
Hardware: RTX 2060
Software: Cutechess-CLI
Time control: 1k nodes/move
Book: openings-10ply-100k.pgn, 10 plies, sequential, color reversed
Code: Select all
# PLAYER : RATING ERROR POINTS PLAYED W L D D(%) CFS(%)
1 lc0.net.62013-tuned : 28.4 23.2 108.0 200 51 35 114 57 97
2 J13B.2-188 : 0.0 14.6 192.5 400 74 89 237 59 54
3 lc0.net.62013 : -1.8 24.4 99.5 200 38 39 123 62 ---
(Big T60 Net. +27 Elo gain)
Re: Lc0: Kiudee setting is very strong
Posted: Mon Jan 20, 2020 4:17 pm
by Master Om
pohl4711 wrote: ↑Mon Jan 20, 2020 2:38 pm
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (
https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Whats Kiudee settings ?
Edit: I got it
Re: Lc0: Kiudee setting is very strong
Posted: Mon Jan 20, 2020 4:19 pm
by OneTrickPony
Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).
Re: Lc0: Kiudee setting is very strong
Posted: Mon Jan 20, 2020 8:05 pm
by mwyoung
pohl4711 wrote: ↑Mon Jan 20, 2020 2:38 pm
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (
https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Thanks, I will give them a try. And I agree that the default settings. Are not the best by a long shot. I never run default with Lc0 in my testing. I will see if this is better. If it is better then my settings. I will use them in my testing.
Re: Lc0: Kiudee setting is very strong
Posted: Mon Jan 20, 2020 8:09 pm
by mwyoung
OneTrickPony wrote: ↑Mon Jan 20, 2020 4:19 pm
Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).
We will have to try the setting and find out. But if it does get you to the same place, but quicker IMO it would still be better then default.
Neural Nets are so much different with the search. All roll outs play to the end of the game. So you always have 100 percent depth. What you are selecting is the width of the search, and what moves you keep and throw away at each "ply" or roll out.
Re: Lc0: Kiudee setting is very strong
Posted: Tue Jan 21, 2020 12:22 am
by Laskos
pohl4711 wrote: ↑Mon Jan 20, 2020 2:38 pm
Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (
https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)
1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)
2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %
Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.
Here the Kiudee setting:
CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:
Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again,
40 Elo points gain is amazing.
Re: Lc0: Kiudee setting is very strong
Posted: Tue Jan 21, 2020 12:47 am
by corres
Laskos wrote: ↑Tue Jan 21, 2020 12:22 am
...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:
Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again,
40 Elo points gain is amazing.
Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
Re: Lc0: Kiudee setting is very strong
Posted: Tue Jan 21, 2020 12:49 am
by Laskos
corres wrote: ↑Tue Jan 21, 2020 12:47 am
Laskos wrote: ↑Tue Jan 21, 2020 12:22 am
...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:
Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match
The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again,
40 Elo points gain is amazing.
Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.