Something goes wrong with lc0 since yesterday?

[jp]

https://docs.google.com/spreadsheets/d/ ... 1707964751

Based on what I see, it seems Leela can't make progress with this net anymore. So, what will happen next ?

I see this doc (MTGOStark) has scaling charts for Lc0 & SF9. Haven't looked much at them yet.

[Milos]

I know that A0 scaled better than stockfish8 and certainly earned elo from more time and it seems that the lc0 team has a serious bug or do something clearly different than A0 or both of them.

A0 scaling vs SF from that crappy paper is totally bogus. It is a fact! I wonder when are ppl gonna stop citing that BS graph...

I prefer to assume they are not lying but if they are lying then what is the reason for them not to tell the public that A0 is good at bullet time control.

It is better to be strong both at bullet time control and at long time control and not only at long time control.

I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .

[Milos]

I also think it's more plausible that we either have some sort of bug that hurts scaling, or simply haven't figured out some part of the AlphaZero approach properly yet. One experiment being tried now is changing the search parameters, i.e. PUCT, FPU reduction and futile move aversion constant. I'm confident that this will eventually be solved, it will just take some time now that this issue has come to light.

"futile move aversion constant" , that's a really funny name. Actually you should call your "changing the search parameters" - "futile search parameters exploration", it is much more suitable name .
Thinking you can magically optimize search with 2 or 3 fixed parameters demonstrates beautifully how clueless everyone there is about chess programming.

[jkiliani]

I also think it's more plausible that we either have some sort of bug that hurts scaling, or simply haven't figured out some part of the AlphaZero approach properly yet. One experiment being tried now is changing the search parameters, i.e. PUCT, FPU reduction and futile move aversion constant. I'm confident that this will eventually be solved, it will just take some time now that this issue has come to light.

"futile move aversion constant" , that's a really funny name. Actually you should call your "changing the search parameters" - "futile search parameters exploration", it is much more suitable name .
Thinking you can magically optimize search with 2 or 3 fixed parameters demonstrates beautifully how clueless everyone there is about chess programming.

And what, oh great wizard of chess, are the right parameters then?

Don't mouth off unless you have something constructive to contribute.

[Milos]

I also think it's more plausible that we either have some sort of bug that hurts scaling, or simply haven't figured out some part of the AlphaZero approach properly yet. One experiment being tried now is changing the search parameters, i.e. PUCT, FPU reduction and futile move aversion constant. I'm confident that this will eventually be solved, it will just take some time now that this issue has come to light.

"futile move aversion constant" , that's a really funny name. Actually you should call your "changing the search parameters" - "futile search parameters exploration", it is much more suitable name .
Thinking you can magically optimize search with 2 or 3 fixed parameters demonstrates beautifully how clueless everyone there is about chess programming.

And what, oh great wizard of chess, are the right parameters then?

Again, believing there are just a few magic parameters is just delusional. Try to learn how does what we call A/B search actually look like, even in simplest A/B engines.
For the starter, averaging operator doesn't work and you can forget about actually improving it for chess.

Don't mouth off unless you have something constructive to contribute.

Why, is it forbidden? What is this, I'm breaking your idealistic harmony, sounding too harsh for your fanboys ears? Breaking your sensitive heart?
May I ask what is your actual contribution to the project, beside cheering and providing "precious" GPU time?

[jp]

I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .

What did your 2.5s correspond to, i.e. how many nps, i.e. what part of the curve?

[Uri Blass]

I know that A0 scaled better than stockfish8 and certainly earned elo from more time and it seems that the lc0 team has a serious bug or do something clearly different than A0 or both of them.

A0 scaling vs SF from that crappy paper is totally bogus. It is a fact! I wonder when are ppl gonna stop citing that BS graph...

I prefer to assume they are not lying but if they are lying then what is the reason for them not to tell the public that A0 is good at bullet time control.

It is better to be strong both at bullet time control and at long time control and not only at long time control.

I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .

I remember reading that A0 lost to stockfish8 at bullet but when the time control became longer it performed better against stockfish.
I do not remember reading about time handicap matches between A0 and itself to test scaling.

[megamau]

I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that.

This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .

Of course this experiment is totally wrong, as it simply assumes the parallel search (multi-thread) to be equivalent to the sequential search (longer timing).
With this kind of reasoning, also Fritz 3 scales perfectly to 128 cores.

[jp]

I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that.

This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .

Of course this experiment is totally wrong, as it simply assumes the parallel search (multi-thread) to be equivalent to the sequential search (longer timing).
With this kind of reasoning, also Fritz 3 scales perfectly to 128 cores.

He's not talking about scaling with cores. He said timing odds. TC was 150s vs 2.5s. 28 threads, which I take as meaning fixed.

[jp]

I don't know about A0 scaling, but if that figure is correct for A0 scaling, it is quite bad. I was talking about SF scaling in that figure. It is simply bogus. SF8 scaling is simply much, much better than that. This thing is relatively easy to check, it just requires time. You run SF8 selfplay with timing odds, like 60s vs 1s. You don't need 64 threads machine. You can just scale time. I was running on 28 threads (with HT) and TC was 150s vs 2.5s. Difference was 250Elo. In that figure difference is less than 50Elo. As I said it is rather trivial to prove that SF8 scaling in that figure is bogus and if A0 scaling is correct it is much worse than SF8 and hoping that Lc0 will somehow have better scaling than A0 is frankly speaking dreaming, but ppl can dream ofc .

I remember reading that A0 lost to stockfish8 at bullet but when the time control became longer it performed better against stockfish.
I do not remember reading about time handicap matches between A0 and itself to test scaling.

Not between A0 and itself. Between A0 and SF8 40ms per move. Meaning SF8 40ms per move all the time. A0 40ms per move up to probably 1 minute per move. And same SF8 vs itself. That's how they made that figure Milos says is bogus.