Lc0: Kiudee setting is very strong

[pohl4711]

Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

[corres]

...
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo.
But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic.
...

Is there "Armageddon Elo" and a method to calculate "Armageddon Elo"?
I do not know about that.
Note
Kiudee-settings may be good for "Armageddon" but what is the situation in the case of normal games?

[pohl4711]

Armageddon means, all draws are counted as a win for Black. And the opening-positions give White an advantage (SALC: White can castle short, Black can castle long, only). Thats it. Because, there are no draws, the results are spreaded (more away from 50%-50%).

Here some tests from discord, with "normal" openings:

Code: Select all

tc=1s+0.1s, RTX 2070
"bonus settings" cpuct=2.147, fpu=0.443, pst=1.607, cpuct-base=18368, cpuct-factor=2.815

   # PLAYER                        :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L
   1 lc0.net.58613.kiudee_bonus    :      86     10   836.5    1422  58.8     100  459  755  208
   2 lc0.net.58613.default         :      33      9   699.0    1422  49.2      55  320  758  344
   3 lc0.net.LD2.kiudee_bonus      :      32     10   696.5    1422  49.0     100  308  777  337
   4 lc0.net.LD2.default           :       0      9   612.0    1422  43.0     ---  240  744  438

(both Nets are small (10x128). +32 and +53 Elo gain = +42 Elo average gain)


Gauntlet: J13B.2-188 vs lc0.net.62013 (default) / lc0.net.62013-tuned @kiudee
LC0-version: lc0-v0.23.2, Backend=cudnn-fp16
Hardware: RTX 2060
Software: Cutechess-CLI
Time control: 1k nodes/move
Book: openings-10ply-100k.pgn, 10 plies, sequential, color reversed

Code: Select all

# PLAYER                 :  RATING  ERROR  POINTS  PLAYED    W    L    D  D(%)  CFS(%)
1 lc0.net.62013-tuned    :    28.4   23.2   108.0     200   51   35  114    57      97
2 J13B.2-188             :     0.0   14.6   192.5     400   74   89  237    59      54
3 lc0.net.62013          :    -1.8   24.4    99.5     200   38   39  123    62     ---

(Big T60 Net. +27 Elo gain)

[Master Om]

Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

Whats Kiudee settings ?
Edit: I got it

[OneTrickPony]

Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).

[mwyoung]

Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

Thanks, I will give them a try. And I agree that the default settings. Are not the best by a long shot. I never run default with Lc0 in my testing. I will see if this is better. If it is better then my settings. I will use them in my testing.

[mwyoung]

Any idea if it still holds for very long TCs or does it converge to the same thing given enough time (which would be my intuition).

We will have to try the setting and find out. But if it does get you to the same place, but quicker IMO it would still be better then default.

Neural Nets are so much different with the search. All roll outs play to the end of the game. So you always have 100 percent depth. What you are selecting is the width of the search, and what moves you keep and throw away at each "ply" or roll out.

[Laskos]

Played 2 Gauntlets a 300 games (30''+300ms Bullet, with my 150 SALC Armageddon openings of my longtime-Testruns (https://www.sp-cc.de/nn-longtime-testing.htm)):
Lc0 default Leelenstein 13 Net vs. Lc0 Kiudee Leelenstein 13 Net and
Lc0 default Leelenstein 13 Net vs. Lc0 LSbinary Leelenstein 13 Net (the binary from josh patreon-site (post from 2019/12/26)

1 Lc0 0.23.2kiudee LS13 vs. Lc0 default LS13: 300 (+180,= 0,-120), 60.0 % (!!!)

2 Lc0 LSbinary LS13 vs. Lc0 default LS13: 300 (+103,= 0,-197), 34.3 %

Conclusions:
Josh-binary is very bad - do not use it!
Lc0 Kiudee is really impressive. 60%-40% means +70 Elo. But mention, that Armageddon (no draws, because all draws are counted as a win for Black) and Bullet-speed spread results, so +40 or +50 Elo seems more realistic. And on discord, some tests with other net-sizes (10x128 and T60) (Leelenstein Size is 20x256) show a measureable Elo gain with Kiudee-setting, too. So, it seems, that the Kiudee-setting should be the new default for Lc0. I will use it from now as default for my Lc0-testings.

Here the Kiudee setting:

CPuct=2.147
Fpu=0.443
PolicyTemperature=1.607
CPuctBase=18368
CPuctFactor=2.815

Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

The engine is Lc0 v0.23.2.

With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:



So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.

Again, 40 Elo points gain is amazing.

[corres]

...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:

So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.

Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.

[Laskos]

...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:

Code: Select all

Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118  [0.555] 200
Elo difference: 38.37 +/- 30.78
Finished match

The engine is Lc0 v0.23.2.
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:

So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.

Is our reported result an "Armageddon" result or a normal, really experienced result?
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.

No, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.