Without "Armageddon"?Laskos wrote: ↑Tue Jan 21, 2020 12:49 amNo, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.corres wrote: ↑Tue Jan 21, 2020 12:47 amIs our reported result an "Armageddon" result or a normal, really experienced result?Laskos wrote: ↑Tue Jan 21, 2020 12:22 am ...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:The engine is Lc0 v0.23.2.Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200 Elo difference: 38.37 +/- 30.78 Finished match
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
Lc0: Kiudee setting is very strong
Moderators: hgm, Rebel, chrisw
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: Lc0: Kiudee setting is very strong
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0: Kiudee setting is very strong
No, no, normal games and scoring. Armageddon is a bit different chess, I am not testing with it as of yet. Unbalanced openings are almost as good as resolving power goes, but it's actual chess and scoring.corres wrote: ↑Tue Jan 21, 2020 12:58 amWithout "Armageddon"?Laskos wrote: ↑Tue Jan 21, 2020 12:49 amNo, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.corres wrote: ↑Tue Jan 21, 2020 12:47 amIs our reported result an "Armageddon" result or a normal, really experienced result?Laskos wrote: ↑Tue Jan 21, 2020 12:22 am ...
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:The engine is Lc0 v0.23.2.Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200 Elo difference: 38.37 +/- 30.78 Finished match
With unbalanced openings in paired games one should use the pentanomial error margins, which are smaller than the usual trinomial ones. Here is how a result looks in my case:
So, all in all about 40 +/- 20 (95% confidence) Elo points improvement, which is amazing. I guess the optimal CPuct will increase with longer time control, but I have no resources to tune it to various TC.
Again, 40 Elo points gain is amazing.
Using unbalanced start positions is not equivalent to using "Armageddon" method to modify the result of games.
-
- Posts: 3657
- Joined: Wed Nov 18, 2015 11:41 am
- Location: hungary
Re: Lc0: Kiudee setting is very strong
Thanks for the answers.Laskos wrote: ↑Tue Jan 21, 2020 1:01 amNo, no, normal games and scoring. Armageddon is a bit different chess, I am not testing with it as of yet. Unbalanced openings are almost as good as resolving power goes, but it's actual chess and scoring.corres wrote: ↑Tue Jan 21, 2020 12:58 amWithout "Armageddon"?Laskos wrote: ↑Tue Jan 21, 2020 12:49 amNo, regular 3-mover human openings, somewhat unbalanced to avoid 90% draw rates. From games of above 2200 FIDE Elo points humans.
-
- Posts: 74
- Joined: Wed Dec 04, 2019 11:25 am
- Full name: Prasanna Bandihole
Re: Lc0: Kiudee setting is very strong
Has anyone observed improvement with regard to endgame play by Lc0? Does the Kiudee settings avoid Lc0 trolling in endgames?
My prediction is that, in coming few months, with T60 NN reaching elo 3000 mark and with settings that help improve endgame play quality, Lc0 will be unbeatable!
My prediction is that, in coming few months, with T60 NN reaching elo 3000 mark and with settings that help improve endgame play quality, Lc0 will be unbeatable!
-
- Posts: 2439
- Joined: Sat Sep 03, 2011 7:25 am
- Location: Berlin, Germany
- Full name: Stefan Pohl
Re: Lc0: Kiudee setting is very strong
On Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!Laskos wrote: ↑Tue Jan 21, 2020 12:22 am
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:
Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200 Elo difference: 38.37 +/- 30.78 Finished match
I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.
https://www.sp-cc.de/nn-longtime-testing.htm
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0: Kiudee setting is very strong
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.pohl4711 wrote: ↑Tue Jan 21, 2020 9:05 amOn Twitch (potato_chess), 2 testruns with Net 62038 were played (1'+2'' on RTX, which is not that short and 62038 is a bigger Net (24x320)). 200 games each. And with Kiudee, the result was +36 Elo better, than the default setting. So, in each testrun, I have seen, Kiudee scored at least +30 Elo better. Nice!Laskos wrote: ↑Tue Jan 21, 2020 12:22 am
Thanks for the info, I can confirm at 30'' + 0.3'' in 200 games from unbalanced openings:
Code: Select all
Score of SV_256x20_1541_Kiudee vs SV_256x20_1541: 52 - 30 - 118 [0.555] 200 Elo difference: 38.37 +/- 30.78 Finished match
I will repeat my longtime-testrun (8'+5'') of Lc0 0.23.1 LS 12.2 with Kiudee-setting. Lets see, if there is an Elo-gain with long thinkingtime, too...but this will take 6 days from now.
https://www.sp-cc.de/nn-longtime-testing.htm
Thanks for posting them, now I will modify Cutechess json file for Lc0 0232 with different nets, all to take these parameters.
-
- Posts: 2439
- Joined: Sat Sep 03, 2011 7:25 am
- Location: Berlin, Germany
- Full name: Stefan Pohl
Re: Lc0: Kiudee setting is very strong
Kiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).Laskos wrote: ↑Tue Jan 21, 2020 10:23 am
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"
https://cdn.discordapp.com/attachments/ ... 26-339.png
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0: Kiudee setting is very strong
Wow, nice. I think something like CLOP would work badly with so many parameters to tune and still having an acceptable time control in that many games. Imagine a fine tuned Lc0, as much fine tuned as Stockfish is.pohl4711 wrote: ↑Tue Jan 21, 2020 10:40 amKiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).Laskos wrote: ↑Tue Jan 21, 2020 10:23 am
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"
https://cdn.discordapp.com/attachments/ ... 26-339.png
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Lc0: Kiudee setting is very strong
pohl4711 wrote: ↑Tue Jan 21, 2020 10:40 amKiudee settings are done by tuning-runs (Kiudee is the nickname of the author on discord). At the moment, he tunes the ShortSightedness parameter (all other parameters are set to the known Kiudee-setting, we test here).Laskos wrote: ↑Tue Jan 21, 2020 10:23 am
Good! 1 + 2'' is already Blitz (large increment), so the optimum CPuct value is probably only smoothly increasing to LTC. Thanks for your test at 8' + 5'' in as many as 300 games, a huge task for a single machine. But it is needed, as these Kiudee parameters give a hard to believe boost. In the past I tried to fiddle with Lc0 engine parameters, it was always within even 1 SD error margins, it is hard to have many games at acceptable TC with Leela. Besides that, I do not understand what many of these parameters do precisely, and just blindly trying here and there is useless.
https://discordapp.com/channels/4254194 ... 8982059008
"Started a small tuning run for ShortSightedness. Same TC as my other tunes, with kiudee-bonus applied. Network: 58613.
Is there a plausible reason for why there is a bump in the middle?"
https://cdn.discordapp.com/attachments/ ... 26-339.png
Against Stockfish 11 the same 40 Elo points difference. And I have never seen at any time in the past in any conditions Stockfish_dev beaten so badly by Leela (above 100 Elo points difference) on my computer. Time control is short Blitz. Normal games and scoring (non-Armageddon).
Code: Select all
RTX 2070 GPU
Fast 4 core i7 CPU
About ~2 Leela Ratio
TC: 2' + 1.2''
Score of SV_256x20_1541 vs SF_11: 36 - 18 - 46 [0.590] 100
Elo difference: 63.23 +/- 50.45
Finished match
Score of SV_256x20_1541_Kiudee vs SF_11: 42 - 13 - 45 [0.645] 100
Elo difference: 103.73 +/- 51.13
Finished match
Remarkable results.
-
- Posts: 6442
- Joined: Tue Jan 09, 2007 12:31 am
- Location: PA USA
- Full name: Louis Zulli
Re: Lc0: Kiudee setting is very strong
Decided to try these settings, just for fun. Lc0 crushes the Fish!
[pgn][White "Lc0 (384x30-t40-swa-1705000)"]
[Black "Stockfish 11 (20 threads)"]
[Result "1-0"]
[TimeControl "180s+3s"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. Be3 e5 7. Nb3 Be7 8. f3
Be6 9. Qd2 O-O 10. O-O-O Nbd7 11. g4 b5 12. g5 Nh5 13. Nd5 Bxd5 14. exd5 f5 15.
Rg1 f4 16. Bf2 Bxg5 17. Bh3 Be7 18. Na5 Kh8 19. Nc6 Qe8 20. Kb1 Nhf6 21. Be6
Bd8 22. Qb4 Bc7 23. a4 Qh5 24. Rd3 Qxh2 25. Qe1 Rae8 26. Qf1 e4 27. Rh1 Qxh1
28. Qxh1 exd3 29. cxd3 Nc5 30. Bxc5 dxc5 31. Qg1 Bd6 32. a5 b4 33. d4 cxd4 34.
Ka2 Ra8 35. Qxd4 Ne8 36. Qc4 Nc7 37. Bd7 Rf7 38. Be6 Rf6 39. Nd4 Be5 40. Kb3
Rh6 41. Bf5 Bxd4 42. Qxc7 Bf6 43. Qf7 Be5 44. Qe7 Bf6 45. Qb7 Rd8 46. Qxa6 Rh2
47. Bc2 Rd2 48. Qe6 R2xd5 49. a6 Bd4 50. Qe7 R5d7 51. Qh4 h6 52. Qxf4 Re7 53.
Qf5 Kg8 54. Be4 Bf2 55. Kxb4 Red7 56. Ka5 Rd6 57. b4 g6 58. Qe5 Bb6+ 59. Kb5
Bd4 60. Qe7 R8d7 61. Qe8+ Kg7 62. Bxg6 Ra7 63. Be4 Rdxa6 64. Bd5 Ba1 65. f4 Kf6
66. Qf8+ Kg6 67. Be4+ Kh5 68. Bf5 Kh4 69. Qc5 Kg3 70. Qe3+ Kg2 71. Bh3+ Kh2 72.
Bf1 Ra2 73. Qh3+ Kg1 74. Bc4 Bf6 75. Bxa2 Rb7+ 76. Kc6 Rg7 77. b5 Bd4 78. b6
Rg6+ 79. Kd5 Bf2 80. b7 Ba7 81. f5 Rg2 82. Qxh6 Rxa2 83. Qb6+ Bxb6 84. b8=Q
Rd2+ 85. Kc6 Bd8 86. f6 Bxf6 87. Qb1+ Kf2 88. Qf5+ Ke3 89. Qxf6 Rd4 90. Qg6 Ra4
91. Qg3+ Ke2 92. Qc3 Rg4 93. Kd5 Rh4 94. Qc2+ Kf3 95. Qd3+ Kf2 96. Ke5 Rh8 97.
Qa3 Rh5+ 98. Kf4 Rh4+ 99. Kf5 Rh7 100. Qb2+ Ke3 101. Qe5+ Kf3 102. Qd5+ Ke3
103. Kf6 Rh8 104. Kg5 Re8 105. Qc5+ Kd2 106. Qa5+ Kc1 107. Qc7+ Kb1 108. Qb7+
Ka1 109. Qa6+ Kb1 110. Qb5+ Ka1 111. Qxe8 Kb2 112. Kf4 Kc3 113. Ke4 Kb4 114.
Kd4 Kb3 115. Qe2 Kb4 116. Qb2+ Ka5 117. Kc5 Ka4 118. Qb4# 1-0[/pgn]
[pgn][White "Lc0 (384x30-t40-swa-1705000)"]
[Black "Stockfish 11 (20 threads)"]
[Result "1-0"]
[TimeControl "180s+3s"]
1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 a6 6. Be3 e5 7. Nb3 Be7 8. f3
Be6 9. Qd2 O-O 10. O-O-O Nbd7 11. g4 b5 12. g5 Nh5 13. Nd5 Bxd5 14. exd5 f5 15.
Rg1 f4 16. Bf2 Bxg5 17. Bh3 Be7 18. Na5 Kh8 19. Nc6 Qe8 20. Kb1 Nhf6 21. Be6
Bd8 22. Qb4 Bc7 23. a4 Qh5 24. Rd3 Qxh2 25. Qe1 Rae8 26. Qf1 e4 27. Rh1 Qxh1
28. Qxh1 exd3 29. cxd3 Nc5 30. Bxc5 dxc5 31. Qg1 Bd6 32. a5 b4 33. d4 cxd4 34.
Ka2 Ra8 35. Qxd4 Ne8 36. Qc4 Nc7 37. Bd7 Rf7 38. Be6 Rf6 39. Nd4 Be5 40. Kb3
Rh6 41. Bf5 Bxd4 42. Qxc7 Bf6 43. Qf7 Be5 44. Qe7 Bf6 45. Qb7 Rd8 46. Qxa6 Rh2
47. Bc2 Rd2 48. Qe6 R2xd5 49. a6 Bd4 50. Qe7 R5d7 51. Qh4 h6 52. Qxf4 Re7 53.
Qf5 Kg8 54. Be4 Bf2 55. Kxb4 Red7 56. Ka5 Rd6 57. b4 g6 58. Qe5 Bb6+ 59. Kb5
Bd4 60. Qe7 R8d7 61. Qe8+ Kg7 62. Bxg6 Ra7 63. Be4 Rdxa6 64. Bd5 Ba1 65. f4 Kf6
66. Qf8+ Kg6 67. Be4+ Kh5 68. Bf5 Kh4 69. Qc5 Kg3 70. Qe3+ Kg2 71. Bh3+ Kh2 72.
Bf1 Ra2 73. Qh3+ Kg1 74. Bc4 Bf6 75. Bxa2 Rb7+ 76. Kc6 Rg7 77. b5 Bd4 78. b6
Rg6+ 79. Kd5 Bf2 80. b7 Ba7 81. f5 Rg2 82. Qxh6 Rxa2 83. Qb6+ Bxb6 84. b8=Q
Rd2+ 85. Kc6 Bd8 86. f6 Bxf6 87. Qb1+ Kf2 88. Qf5+ Ke3 89. Qxf6 Rd4 90. Qg6 Ra4
91. Qg3+ Ke2 92. Qc3 Rg4 93. Kd5 Rh4 94. Qc2+ Kf3 95. Qd3+ Kf2 96. Ke5 Rh8 97.
Qa3 Rh5+ 98. Kf4 Rh4+ 99. Kf5 Rh7 100. Qb2+ Ke3 101. Qe5+ Kf3 102. Qd5+ Ke3
103. Kf6 Rh8 104. Kg5 Re8 105. Qc5+ Kd2 106. Qa5+ Kc1 107. Qc7+ Kb1 108. Qb7+
Ka1 109. Qa6+ Kb1 110. Qb5+ Ka1 111. Qxe8 Kb2 112. Kf4 Kc3 113. Ke4 Kb4 114.
Kd4 Kb3 115. Qe2 Kb4 116. Qb2+ Ka5 117. Kc5 Ka4 118. Qb4# 1-0[/pgn]