Milos wrote: ↑Tue Jun 05, 2018 2:01 pm
That looks ok but I am confused which options did you use, since difference between yours June 4 and "default" June 2 version is basically just FPUR increased from 0.2 to 0.9, which is totally opposite direction from before and very mild increase of PCUT from 3.1 to 3.4 or something.
Anyway difference is still within error margins so could be just luck or it just worked against particular engine. What I've thought you did is to test it against default LC0 settings for PCUT and FPUG which are for cudnn version something like 1.7 and 0.0.
No, there is a third you did not notice which has a similarly strong effect: Policy Softmax Temperature. I myself did not know it until its author brought it to my attention in Discord.
Well Alex is obviously playing with search and randomly introducing new parameters. This is clearly not a parameter that is either present in LCzero nor A0, nor any standard UCT papers.
I'm not saying that it might not help, but just randomly introducing search parameters is certainly not the best way to develop something.
Also since you are just playing with search parameters I really see no point in playing small amount of matches against random opponents. Much more useful and scientific would be SF testing approach, meaning playing large number of games new version vs. master, and using proper stopping criteria.
Milos wrote: ↑Tue Jun 05, 2018 2:01 pm
That looks ok but I am confused which options did you use, since difference between yours June 4 and "default" June 2 version is basically just FPUR increased from 0.2 to 0.9, which is totally opposite direction from before and very mild increase of PCUT from 3.1 to 3.4 or something.
Anyway difference is still within error margins so could be just luck or it just worked against particular engine. What I've thought you did is to test it against default LC0 settings for PCUT and FPUG which are for cudnn version something like 1.7 and 0.0.
No, there is a third you did not notice which has a similarly strong effect: Policy Softmax Temperature. I myself did not know it until its author brought it to my attention in Discord.
Well Alex is obviously playing with search and randomly introducing new parameters. This is clearly not a parameter that is either present in LCzero nor A0, nor any standard UCT papers.
I'm not saying that it might not help, but just randomly introducing search parameters is certainly not the best way to develop something.
Also since you are just playing with search parameters I really see no point in playing small amount of matches against random opponents. Much more useful and scientific would be SF testing approach, meaning playing large number of games new version vs. master, and using proper stopping criteria.
Actually, it is not him, and was introduced into LCZero. No one had bothered using it or exploring its impact, so I did. You are right that it is a hacky bandaid, and once they are able to sort things out, should not be needed.
"Tactics are the bricks and sticks that make up a game, but positional play is the architectural blueprint."
Albert Silver wrote: ↑Mon Jun 04, 2018 9:02 pm
In view of the many settings in LC0, I ran a deep CLOP on three big ones at the same time. CLOP is designed to tune for the best playing performance, not best tactics or anything. The old default settings were nearly catastrophic giving NN369 a mere 109/200 solved in my corrected WAC set. The new settings yield... 159/200.
You can try them without any work, by downloading the latest LC0 June 4 build at https://crem.xyz/lc0/ dubbed experimental.
Congratulations, seems to work at Longer TC too:
10'+ 10'' LC0 on GTX 1060 and Komodo 10.2 on 1 core (3369 Elo CCRL points 40/4').
Score of LC0_exp_NN373 vs Komodo 10.2: 3 - 6 - 11 [0.425]
Elo difference: -52.51 +/- 105.00
20 of 20 games finished.
+3 -6 =11
8.5/20
Default: +1 -14 =5
3.5/20
My "extreme tactical settings": +0 -13 =7
3.5/20
The performance is more than 200 Elo points better, the samples are small, but there is still 97%-98% confidence that "experimental" performs better in these conditions than each of them. By the way, the performance here is about 3300 CCRL Elo points at longer TC.
Score of LC0_exp_NN373 vs Houdini 1.5a: 49 - 27 - 24 [0.610]
Elo difference: 77.71 +/- 60.99
100 of 100 games finished.
+49 -27 =24
61.0/100
Default:
+37 -33 =30
52.0/100
More than 60 Elo points improvement, almost outside error margins. Performance LC0_exp NN373 of about 3240 CCRL Elo points at 1'+ 1'' and 3320 CCRL Elo points at 10'+ 10'', so it seems to scale well too.
