ZZ-tune conclusively better than the Kiudee default for Lc0

[Laskos]

I saw in the Lc0 discord the ZZ-tune for Lc0 obtained using the chess-tuning-tools software. The tuned parameters are:

Code: Select all

                           Kiudee
                  ZZ-tune  default
CPuct              1.350   2.147
FpuValue           0.228   0.443
PolicyTemperature  1.497   1.607
CPuctFactor        3.468   2.147
CPuctBase          20253   18368

It is hoped that this tune improves upon the default one when using T60, T70, SV and JH nets. Using the chess-tuning-tools, the results of the tune look as following charts:





and



In the last plot (Compute_CPuct), we see that the ZZ-tune is quite neutral in the region of 30 000 nodes per move (npm), gaining in importance with larger npm or TC. If I understood it, the gain from ZZ-tune, especially to larger npm (TC), is unclear. I have three matches: at about 20k npm, 100k npm and 400k npm using the last T70 net 703810 in self-play. I am using unbalanced openings and computing the correct pentanomial error margins, which are much smaller here than the naive trinomial ones shown in Cutechess-Cli. Computing the Normalized Elo too, which shows the scaling of the possible gain at different time controls or npm. The GPU is RTX 2070. The results of the three matches are here:

Code: Select all

20 000 nodes per move
Score of T70_ZZ vs T70_Kiudee: 79 - 74 - 247  [0.506] 400
...      T70_ZZ playing White: 75 - 8 - 117  [0.667] 200
...      T70_ZZ playing Black: 4 - 66 - 130  [0.345] 200
...      White vs Black: 141 - 12 - 247  [0.661] 400
Elo difference: 4.3 +/- 21.1, LOS: 65.7 %, DrawRatio: 61.8 %
Finished match
4.3 +/- 8.5
Normalized Elo: 0.025 +/- 0.050
Low nmp, inconclusive

Code: Select all

100 000 nodes per move
Score of T70_ZZ vs T70_Kiudee: 74 - 50 - 276  [0.530] 400
...      T70_ZZ playing White: 69 - 2 - 129  [0.667] 200
...      T70_ZZ playing Black: 5 - 48 - 147  [0.393] 200
...      White vs Black: 117 - 7 - 276  [0.637] 400
Elo difference: 20.9 +/- 18.9, LOS: 98.4 %, DrawRatio: 69.0 %
Finished match
20.9 +/- 8.04
Normalized Elo: 0.130 +/- 0.050 
Large npm, conclusive

Code: Select all

400 000 nodes per move
Score of T70_ZZ vs T70_Kiudee: 23 - 13 - 64  [0.550] 100
...      T70_ZZ playing White: 23 - 0 - 27  [0.730] 50
...      T70_ZZ playing Black: 0 - 13 - 37  [0.370] 50
...      White vs Black: 36 - 0 - 64  [0.680] 100
Elo difference: 34.9 +/- 41.1, LOS: 94.9 %, DrawRatio: 64.0 %
Finished match
34.9 +/- 14.9
Normalized Elo: 0.236 +/- 0.100
Very large npm, conclusive

We can conclude with fairly high dose of confidence (above 95%) that the gain due to ZZ-tune compared to the defaults increases with time control or nodes per move. Lc0 using ZZ-tune parameters scales better to LTC and the gain compared to the defaults can likely exceed 30 Elo points. It is plausible one will have similar results to similar npm with T60 nets and JH nets. Checking directly the gain to 400k nodes npm with 30x384 nets is prohibitively compute expensive, we have to assume that these T70 results translate to larger nets with similar gains.

[Pi4Chess]

May be this belongs more in Programming and technical discussions

[Zenmastur]

May be this belongs more in Programming and technical discussions

i'M NOT SO SURE ABOUT THAT THE PEOPLE THAT ARE LIKELY TO USE IT....many many are right here....

Good work Kai!!!

+2

Regards,

Zenmastur

[MMarco]

Code: Select all

400 000 nodes per move
Score of T70_ZZ vs T70_Kiudee: 23 - 13 - 64  [0.550] 100
...      T70_ZZ playing White: 23 - 0 - 27  [0.730] 50
...      T70_ZZ playing Black: 0 - 13 - 37  [0.370] 50
...      White vs Black: 36 - 0 - 64  [0.680] 100
Elo difference: 34.9 +/- 41.1, LOS: 94.9 %, DrawRatio: 64.0 %
Finished match
34.9 +/- 14.9
Normalized Elo: 0.236 +/- 0.100
Very large npm, conclusive

We can conclude with fairly high dose of confidence (above 95%) that the gain due to ZZ-tune compared to the defaults increases with time control or nodes per move. Lc0 using ZZ-tune parameters scales better to LTC and the gain compared to the defaults can likely exceed 30 Elo points. It is plausible one will have similar results to similar npm with T60 nets and JH nets. Checking directly the gain to 400k nodes npm with 30x384 nets is prohibitively compute expensive, we have to assume that these T70 results translate to larger nets with similar gains.

Very nice!

I too hope that this translate into larger nets. I quickly double-checked your results with a 48x5 net (from dkappe: https://github.com/dkappe/leela-chess-w ... d-Networks). Games were played at 100s + 1s where the average thinking time is a bit more than 2s per move. On my GTX 1660 Ti with this small distilled net, Lc0 26.3 reaches about 100 knps in a typical early middlegame position without tree reuse (so more than 200k nodes per move for a bit more than 2s per move). With tree reuse, I would say the average npm could be around 400k or even above (from visual inspection, looking at the output during the games). When there a few consecutive ponder hits, the node count went well above 500k per move (with tree reuse), and sometimes further than 1000k per move. I used a well-tested, balanced opening book, Balsa_v500.pgn. After 50 games, the score is the following:

Code: Select all

Score of lc0-263-48x5-zz vs lc0-263-48x5: 8 - 2 - 40 [0.560]
Elo difference: 41.9 +/- 42.3, LOS: 97.1 %, DrawRatio: 80.0 %

50 of 1000 games finished.

I'll push the match further, but these early results confirm your findings.

Here is the lc0.config file used:

--weights=11258-48x5-se.pb.gz
--backend=cudnn-fp16
--nncache=2000000
--threads=2
--cpuct=1.35
--cpuct-at-root=1.35
--cpuct-base=20253
--cpuct-base-at-root=20253
--cpuct-factor=3.468
--cpuct-factor-at-root=3.468
--fpu-value=0.228
--policy-softmax-temp=1.497
--syzygy-paths=C:\syzygy

[Laskos]

Code: Select all

400 000 nodes per move
Score of T70_ZZ vs T70_Kiudee: 23 - 13 - 64  [0.550] 100
...      T70_ZZ playing White: 23 - 0 - 27  [0.730] 50
...      T70_ZZ playing Black: 0 - 13 - 37  [0.370] 50
...      White vs Black: 36 - 0 - 64  [0.680] 100
Elo difference: 34.9 +/- 41.1, LOS: 94.9 %, DrawRatio: 64.0 %
Finished match
34.9 +/- 14.9
Normalized Elo: 0.236 +/- 0.100
Very large npm, conclusive

We can conclude with fairly high dose of confidence (above 95%) that the gain due to ZZ-tune compared to the defaults increases with time control or nodes per move. Lc0 using ZZ-tune parameters scales better to LTC and the gain compared to the defaults can likely exceed 30 Elo points. It is plausible one will have similar results to similar npm with T60 nets and JH nets. Checking directly the gain to 400k nodes npm with 30x384 nets is prohibitively compute expensive, we have to assume that these T70 results translate to larger nets with similar gains.

Very nice!

I too hope that this translate into larger nets. I quickly double-checked your results with a 48x5 net (from dkappe: https://github.com/dkappe/leela-chess-w ... d-Networks). Games were played at 100s + 1s where the average thinking time is a bit more than 2s per move. On my GTX 1660 Ti with this small distilled net, Lc0 26.3 reaches about 100 knps in a typical early middlegame position without tree reuse (so more than 200k nodes per move for a bit more than 2s per move). With tree reuse, I would say the average npm could be around 400k or even above (from visual inspection, looking at the output during the games). When there a few consecutive ponder hits, the node count went well above 500k per move (with tree reuse), and sometimes further than 1000k per move. I used a well-tested, balanced opening book, Balsa_v500.pgn. After 50 games, the score is the following:

Code: Select all

Score of lc0-263-48x5-zz vs lc0-263-48x5: 8 - 2 - 40 [0.560]
Elo difference: 41.9 +/- 42.3, LOS: 97.1 %, DrawRatio: 80.0 %

50 of 1000 games finished.

I'll push the match further, but these early results confirm your findings.

Here is the lc0.config file used:

--weights=11258-48x5-se.pb.gz
--backend=cudnn-fp16
--nncache=2000000
--threads=2
--cpuct=1.35
--cpuct-at-root=1.35
--cpuct-base=20253
--cpuct-base-at-root=20253
--cpuct-factor=3.468
--cpuct-factor-at-root=3.468
--fpu-value=0.228
--policy-softmax-temp=1.497
--syzygy-paths=C:\syzygy

Yes, these are the ZZ-tune parameters.

Second run of 100 games of this 400k npm test is somewhat less spectacular, but it still shows advantage for ZZ-tune at this LTC:

Code: Select all

Score of T70_ZZ vs T70_Kiudee: 16 - 10 - 74  [0.530] 100
...      T70_ZZ playing White: 16 - 0 - 34  [0.660] 50
...      T70_ZZ playing Black: 0 - 10 - 40  [0.400] 50
...      White vs Black: 26 - 0 - 74  [0.630] 100
Elo difference: 20.9 +/- 34.7, LOS: 88.0 %, DrawRatio: 74.0 %
Finished match

Combined the two 100 games matches give 27.9 +/- 11.2 (1SD) Elo points advantage for ZZ-tune in 200 games. Normalized Elo = 0.176 +/- 0.070

[Laskos]

40 games with J92-330 at 120k npm

Code: Select all

Score of J92_ZZ vs J92_Kiudee: 5 - 3 - 32  [0.525] 40
...      J92_ZZ playing White: 5 - 0 - 15  [0.625] 20
...      J92_ZZ playing Black: 0 - 3 - 17  [0.425] 20
...      White vs Black: 8 - 0 - 32  [0.600] 40
Elo difference: 17.4 +/- 48.2, LOS: 76.0 %, DrawRatio: 80.0 %
Finished match

Not very conclusive and at this LTC in self-play even my usual unbalanced openings are too drawish. Another 40 games now with a bit more unbalanced openings.

[corres]

I use Kiudee and ZZ params also for may TRIAD and I did not see important difference between them.
The competition on CCC also does not show great enhancement in Leela chess power.
For the simplicity I made speed tests with my TRIAD and J92-330 net
Test 1:
with Kiudee params max.speed= 44.1 Kn/sec (Depth = 14, Time = 147 sec, Nodes N = 6.2 Mn)
Test 2:
with ZZ params max.speed= 47.5 Kn/sec (Depth = 14, T = 128 sec, Nodes N = 5.9 Mn)
Other params
Threads = 6
Backend = Multiplexing
BackendOptions = (backend=cuda-fp16,gpu=0),(backend=cuda-fp16,gpu=1),(backend=cuda-fp16,gpu=2)
NNCacheSize = 20000000
MinibatchSize = 256
Others are Leela default.
The speed enhancement is about 8 % so we can not wait too many from ZZ tuning either.
Maybe the ZZ tuning repair Leela tactical ability in some measure.
I used CUDA ver 11.1 and the last NVIDIA drv.

[MMarco]

I finally let it run to 700 games. It was a perfect tie in the last 650 games. The surprisingly strong start for zz-tune couldn't be maintained, and the results are now inconclusive (though zz-tune doesn't appear weaker).

100s + 1s, GTX 1660 Ti, Balsa_v500.pgn

Code: Select all

   # PLAYER             :  RATING  ERROR  PLAYED    (%)   CFS    W    D    L   D(%)
   1 lc0-263-48x5-zz    :     3.0   10.8     700  50.43    71   59  588   53  84.00
   2 lc0-263-48x5       :     0.0   ----     700  49.57   ---   53  588   59  84.00

White advantage = 31.13 +/- 5.52
Draw rate (equal opponents) = 86.33 % +/- 1.34

[Laskos]

I finally let it run to 700 games. It was a perfect tie in the last 650 games. The surprisingly strong start for zz-tune couldn't be maintained, and the results are now inconclusive (though zz-tune doesn't appear weaker).

100s + 1s, GTX 1660 Ti, Balsa_v500.pgn

Code: Select all

   # PLAYER             :  RATING  ERROR  PLAYED    (%)   CFS    W    D    L   D(%)
   1 lc0-263-48x5-zz    :     3.0   10.8     700  50.43    71   59  588   53  84.00
   2 lc0-263-48x5       :     0.0   ----     700  49.57   ---   53  588   59  84.00

White advantage = 31.13 +/- 5.52
Draw rate (equal opponents) = 86.33 % +/- 1.34

Thanks, that's disappointing. I have a total of 160 time consuming games now with J92-330 at 120k npm, which in total show some 20-23 Elo points advantage for ZZ-tune, but still not very conclusive even using pentanomial variance.

Code: Select all

Score of J92_ZZ vs J92_Kiudee: 5 - 3 - 32  [0.525] 40
...      J92_ZZ playing White: 5 - 0 - 15  [0.625] 20
...      J92_ZZ playing Black: 0 - 3 - 17  [0.425] 20
...      White vs Black: 8 - 0 - 32  [0.600] 40
Elo difference: 17.4 +/- 48.2, LOS: 76.0 %, DrawRatio: 80.0 %
Finished match

Score of J92_ZZ vs J92_Kiudee: 6 - 4 - 30  [0.525] 40
...      J92_ZZ playing White: 6 - 0 - 14  [0.650] 20
...      J92_ZZ playing Black: 0 - 4 - 16  [0.400] 20
...      White vs Black: 10 - 0 - 30  [0.625] 40
Elo difference: 17.4 +/- 54.1, LOS: 73.6 %, DrawRatio: 75.0 %
Finished match

Score of J92_ZZ vs J92_Kiudee: 13 - 13 - 14  [0.500] 40
...      J92_ZZ playing White: 13 - 0 - 7  [0.825] 20
...      J92_ZZ playing Black: 0 - 13 - 7  [0.175] 20
...      White vs Black: 26 - 0 - 14  [0.825] 40
Elo difference: 0.0 +/- 88.6, LOS: 50.0 %, DrawRatio: 35.0 %
Finished match

Score of J92_ZZ vs J92_Kiudee: 13 - 7 - 20  [0.575] 40
...      J92_ZZ playing White: 13 - 0 - 7  [0.825] 20
...      J92_ZZ playing Black: 0 - 7 - 13  [0.325] 20
...      White vs Black: 20 - 0 - 20  [0.750] 40
Elo difference: 52.5 +/- 77.3, LOS: 91.0 %, DrawRatio: 50.0 %
Finished match

[MMarco]

Thanks, that's disappointing. I have a total of 160 time consuming games now with J92-330 at 120k npm, which in total show some 20-23 Elo points advantage for ZZ-tune, but still not very conclusive even using pentanomial variance.

Code: Select all

Score of J92_ZZ vs J92_Kiudee: 5 - 3 - 32  [0.525] 40
...      J92_ZZ playing White: 5 - 0 - 15  [0.625] 20
...      J92_ZZ playing Black: 0 - 3 - 17  [0.425] 20
...      White vs Black: 8 - 0 - 32  [0.600] 40
Elo difference: 17.4 +/- 48.2, LOS: 76.0 %, DrawRatio: 80.0 %
Finished match

Score of J92_ZZ vs J92_Kiudee: 6 - 4 - 30  [0.525] 40
...      J92_ZZ playing White: 6 - 0 - 14  [0.650] 20
...      J92_ZZ playing Black: 0 - 4 - 16  [0.400] 20
...      White vs Black: 10 - 0 - 30  [0.625] 40
Elo difference: 17.4 +/- 54.1, LOS: 73.6 %, DrawRatio: 75.0 %
Finished match

Score of J92_ZZ vs J92_Kiudee: 13 - 13 - 14  [0.500] 40
...      J92_ZZ playing White: 13 - 0 - 7  [0.825] 20
...      J92_ZZ playing Black: 0 - 13 - 7  [0.175] 20
...      White vs Black: 26 - 0 - 14  [0.825] 40
Elo difference: 0.0 +/- 88.6, LOS: 50.0 %, DrawRatio: 35.0 %
Finished match

Score of J92_ZZ vs J92_Kiudee: 13 - 7 - 20  [0.575] 40
...      J92_ZZ playing White: 13 - 0 - 7  [0.825] 20
...      J92_ZZ playing Black: 0 - 7 - 13  [0.325] 20
...      White vs Black: 20 - 0 - 20  [0.750] 40
Elo difference: 52.5 +/- 77.3, LOS: 91.0 %, DrawRatio: 50.0 %
Finished match

Well, kind of. For my test, maybe Balsa opening set was too drawish. I started again with Pohl's unbalanced openings (the 664 pos. 4-moves set), still at 100s + 1s. After 100 games zz-tune has a good start:

Code: Select all

Score of lc0-263-48x5-zz vs lc0-263-48x5: 25 - 15 - 60 [0.550]
Elo difference: 34.9 +/- 43.1, LOS: 94.3 %, DrawRatio: 60.0 %

100 of 1328 games finished.

So far what is see is that zz-tune does always come out on top, even if by not much. I think it is still promising. Let's see. I'll try to complete the 1328 games (a bit more than 4 days).