I see this doc (MTGOStark) has scaling charts for Lc0 & SF9. Haven't looked much at them yet.JJJ wrote: ↑Sat Aug 25, 2018 12:24 pm https://docs.google.com/spreadsheets/d/ ... 1707964751
Based on what I see, it seems Leela can't make progress with this net anymore. So, what will happen next ?
Something goes wrong with lc0 since yesterday?
Moderators: hgm, Rebel, chrisw
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Something goes wrong with lc0 since yesterday?
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Something goes wrong with lc0 since yesterday?
I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .Uri Blass wrote: ↑Sat Aug 25, 2018 9:58 amI prefer to assume they are not lying but if they are lying then what is the reason for them not to tell the public that A0 is good at bullet time control.
It is better to be strong both at bullet time control and at long time control and not only at long time control.
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Something goes wrong with lc0 since yesterday?
"futile move aversion constant" , that's a really funny name. Actually you should call your "changing the search parameters" - "futile search parameters exploration", it is much more suitable name .jkiliani wrote: ↑Sat Aug 25, 2018 11:16 am I also think it's more plausible that we either have some sort of bug that hurts scaling, or simply haven't figured out some part of the AlphaZero approach properly yet. One experiment being tried now is changing the search parameters, i.e. PUCT, FPU reduction and futile move aversion constant. I'm confident that this will eventually be solved, it will just take some time now that this issue has come to light.
Thinking you can magically optimize search with 2 or 3 fixed parameters demonstrates beautifully how clueless everyone there is about chess programming.
-
- Posts: 143
- Joined: Wed Jan 17, 2018 1:26 pm
Re: Something goes wrong with lc0 since yesterday?
And what, oh great wizard of chess, are the right parameters then?Milos wrote: ↑Sun Aug 26, 2018 12:06 am"futile move aversion constant" , that's a really funny name. Actually you should call your "changing the search parameters" - "futile search parameters exploration", it is much more suitable name .jkiliani wrote: ↑Sat Aug 25, 2018 11:16 am I also think it's more plausible that we either have some sort of bug that hurts scaling, or simply haven't figured out some part of the AlphaZero approach properly yet. One experiment being tried now is changing the search parameters, i.e. PUCT, FPU reduction and futile move aversion constant. I'm confident that this will eventually be solved, it will just take some time now that this issue has come to light.
Thinking you can magically optimize search with 2 or 3 fixed parameters demonstrates beautifully how clueless everyone there is about chess programming.
Don't mouth off unless you have something constructive to contribute.
-
- Posts: 4190
- Joined: Wed Nov 25, 2009 1:47 am
Re: Something goes wrong with lc0 since yesterday?
Again, believing there are just a few magic parameters is just delusional. Try to learn how does what we call A/B search actually look like, even in simplest A/B engines.jkiliani wrote: ↑Sun Aug 26, 2018 12:30 amAnd what, oh great wizard of chess, are the right parameters then?Milos wrote: ↑Sun Aug 26, 2018 12:06 am"futile move aversion constant" , that's a really funny name. Actually you should call your "changing the search parameters" - "futile search parameters exploration", it is much more suitable name .jkiliani wrote: ↑Sat Aug 25, 2018 11:16 am I also think it's more plausible that we either have some sort of bug that hurts scaling, or simply haven't figured out some part of the AlphaZero approach properly yet. One experiment being tried now is changing the search parameters, i.e. PUCT, FPU reduction and futile move aversion constant. I'm confident that this will eventually be solved, it will just take some time now that this issue has come to light.
Thinking you can magically optimize search with 2 or 3 fixed parameters demonstrates beautifully how clueless everyone there is about chess programming.
For the starter, averaging operator doesn't work and you can forget about actually improving it for chess.
Why, is it forbidden? What is this, I'm breaking your idealistic harmony, sounding too harsh for your fanboys ears? Breaking your sensitive heart?Don't mouth off unless you have something constructive to contribute.
May I ask what is your actual contribution to the project, beside cheering and providing "precious" GPU time?
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Something goes wrong with lc0 since yesterday?
What did your 2.5s correspond to, i.e. how many nps, i.e. what part of the curve?Milos wrote: ↑Sat Aug 25, 2018 11:57 pm I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .
-
- Posts: 10296
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: Something goes wrong with lc0 since yesterday?
I remember reading that A0 lost to stockfish8 at bullet but when the time control became longer it performed better against stockfish.Milos wrote: ↑Sat Aug 25, 2018 11:57 pmI don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .Uri Blass wrote: ↑Sat Aug 25, 2018 9:58 amI prefer to assume they are not lying but if they are lying then what is the reason for them not to tell the public that A0 is good at bullet time control.
It is better to be strong both at bullet time control and at long time control and not only at long time control.
I do not remember reading about time handicap matches between A0 and itself to test scaling.
-
- Posts: 37
- Joined: Wed Feb 10, 2016 6:20 am
- Location: Singapore
Re: Something goes wrong with lc0 since yesterday?
Of course this experiment is totally wrong, as it simply assumes the parallel search (multi-thread) to be equivalent to the sequential search (longer timing).Milos wrote: ↑Sat Aug 25, 2018 11:57 pm I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that.
This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .
With this kind of reasoning, also Fritz 3 scales perfectly to 128 cores.
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Something goes wrong with lc0 since yesterday?
He's not talking about scaling with cores. He said timing odds. TC was 150s vs 2.5s. 28 threads, which I take as meaning fixed.megamau wrote: ↑Sun Aug 26, 2018 8:55 amOf course this experiment is totally wrong, as it simply assumes the parallel search (multi-thread) to be equivalent to the sequential search (longer timing).Milos wrote: ↑Sat Aug 25, 2018 11:57 pm I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that.
This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .
With this kind of reasoning, also Fritz 3 scales perfectly to 128 cores.
-
- Posts: 1470
- Joined: Mon Apr 23, 2018 7:54 am
Re: Something goes wrong with lc0 since yesterday?
Not between A0 and itself. Between A0 and SF8 40ms per move. Meaning SF8 40ms per move all the time. A0 40ms per move up to probably 1 minute per move. And same SF8 vs itself. That's how they made that figure Milos says is bogus.Uri Blass wrote: ↑Sun Aug 26, 2018 8:25 amI remember reading that A0 lost to stockfish8 at bullet but when the time control became longer it performed better against stockfish.Milos wrote: ↑Sat Aug 25, 2018 11:57 pm I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .
I do not remember reading about time handicap matches between A0 and itself to test scaling.