Lc0 question

[lkaufman]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

[mwyoung]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

Can you show an example.

[lkaufman]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

Can you show an example.

Well, it's easy to get the problematic behavior, it's another thing entirely to get it to replicate in a given position. For example, if I just go thru the main line of the Marshall Gambit in the Spanish, taking moves back now and then, I'll randomly see absurd blunders pop up to the top now and then and go away after a couple seconds. Probably something isn't reset properly on takeback. I can't just give a sequence that shows the problem directly. This is with ChessBase GUI, maybe that's a factor.

[mwyoung]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

Can you show an example.

Well, it's easy to get the problematic behavior, it's another thing entirely to get it to replicate in a given position. For example, if I just go thru the main line of the Marshall Gambit in the Spanish, taking moves back now and then, I'll randomly see absurd blunders pop up to the top now and then and go away after a couple seconds. Probably something isn't reset properly on takeback. I can't just give a sequence that shows the problem directly. This is with ChessBase GUI, maybe that's a factor.

Is this behavior the same regardless of the NN being used? And I also use the Chessbase GUI.

It could be a collision event. What settings are you using? I do not see this behavior in straight games on my GPU.

[lkaufman]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

Can you show an example.

Well, it's easy to get the problematic behavior, it's another thing entirely to get it to replicate in a given position. For example, if I just go thru the main line of the Marshall Gambit in the Spanish, taking moves back now and then, I'll randomly see absurd blunders pop up to the top now and then and go away after a couple seconds. Probably something isn't reset properly on takeback. I can't just give a sequence that shows the problem directly. This is with ChessBase GUI, maybe that's a factor.

Is this behavior the same regardless of the NN being used? And I also use the Chessbase GUI.

It could be a collision event. What settings are you using? I do not see this behavior in straight games on my GPU.

Well, I don't get the same exact behavior on every NN, but yes, it does happen on all NNs sooner or later as far as I can tell. As I mentioned, I raised the "max collision events" from 32 to 256 because I read that some people recommended doing so. Was this perhaps a mistake? Otherwise, I use fp 16, 20 million for NNCacheSize, 3.3 for CPuct, and 10,000 for CPuctBase. Any of this seem suspicious?

[mwyoung]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

Can you show an example.

Well, it's easy to get the problematic behavior, it's another thing entirely to get it to replicate in a given position. For example, if I just go thru the main line of the Marshall Gambit in the Spanish, taking moves back now and then, I'll randomly see absurd blunders pop up to the top now and then and go away after a couple seconds. Probably something isn't reset properly on takeback. I can't just give a sequence that shows the problem directly. This is with ChessBase GUI, maybe that's a factor.

Is this behavior the same regardless of the NN being used? And I also use the Chessbase GUI.

It could be a collision event. What settings are you using? I do not see this behavior in straight games on my GPU.

Well, I don't get the same exact behavior on every NN, but yes, it does happen on all NNs sooner or later as far as I can tell. As I mentioned, I raised the "max collision events" from 32 to 256 because I read that some people recommended doing so. Was this perhaps a mistake? Otherwise, I use fp 16, 20 million for NNCacheSize, 3.3 for CPuct, and 10,000 for CPuctBase. Any of this seem suspicious?

This is the best configuration I have found on my system for straight games. But not back and forth analysis.
You can try it and see if it helps. But it may not if your are rocking the analysis. It may need more testing for your usage in the Chessbase GUI.
Make sure you are using the latest version of Lc0. v21.2. Correction Hash size should be set to 0

best setting for straight games.jpg

[lkaufman]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

Can you show an example.

Well, it's easy to get the problematic behavior, it's another thing entirely to get it to replicate in a given position. For example, if I just go thru the main line of the Marshall Gambit in the Spanish, taking moves back now and then, I'll randomly see absurd blunders pop up to the top now and then and go away after a couple seconds. Probably something isn't reset properly on takeback. I can't just give a sequence that shows the problem directly. This is with ChessBase GUI, maybe that's a factor.

Is this behavior the same regardless of the NN being used? And I also use the Chessbase GUI.

It could be a collision event. What settings are you using? I do not see this behavior in straight games on my GPU.

Well, I don't get the same exact behavior on every NN, but yes, it does happen on all NNs sooner or later as far as I can tell. As I mentioned, I raised the "max collision events" from 32 to 256 because I read that some people recommended doing so. Was this perhaps a mistake? Otherwise, I use fp 16, 20 million for NNCacheSize, 3.3 for CPuct, and 10,000 for CPuctBase. Any of this seem suspicious?

This is the best configuration I have found on my system for straight games. But not back and forth analysis.
You can try it and see if it helps. But it may not if your are rocking the analysis. It may need more testing for your usage in the Chessbase GUI.
Make sure you are using the latest version of Lc0. v21.2. Correction Hash size should be set to 0

best setting for straight games.jpg

OK, I tried most of your settings (I keep threads to 2 to reserve the other 6 for Komodo). I still get the problem, even without takebacks, but perhaps it's because I'm entering the moves of the opening rather quickly in infinite mode, and perhaps it gets confused switching sides so quickly. I don't get the problem when I just start the engine from the position. A couple of settings questions:
1. Do you get a noticeable speedup from 4 threads rather than 2 (or 3)?
2. Is there a reason you keep Cache under 10 million (9,900,000 sounds like an effort to avoid exceeding 10 million)?
3. I don't see any place on my settings page for Hash Size, I am using 21.2, maybe it's due to the specific GUI?
4. Do you change settings based on time control? Perhaps blitz and standard games require rather different settings?
5. Were these settings recommended to you by someone on the Lc0 team, or did you just get them by trial and error?

Thanks.

[mwyoung]

Sometimes when I'm analyzing with Lc0 (infinite mode, 2 cpus, 2080 gpu, multipv display) some silly blunder shows up as the top move, with an appropriately bad score, and may remain on top for several seconds. On rare occasions even two blunders occupy the first two slots for a couple seconds. Is this normal behavior? I use default or recommended settings, I did read somewhere to set max collision events to 256 (rather than 32). Could that be the problem? Anything else I could be doing wrong? The engine sees how bad the moves are, but puts them on top anyway.

Can you show an example.

Well, it's easy to get the problematic behavior, it's another thing entirely to get it to replicate in a given position. For example, if I just go thru the main line of the Marshall Gambit in the Spanish, taking moves back now and then, I'll randomly see absurd blunders pop up to the top now and then and go away after a couple seconds. Probably something isn't reset properly on takeback. I can't just give a sequence that shows the problem directly. This is with ChessBase GUI, maybe that's a factor.

Is this behavior the same regardless of the NN being used? And I also use the Chessbase GUI.

It could be a collision event. What settings are you using? I do not see this behavior in straight games on my GPU.

Well, I don't get the same exact behavior on every NN, but yes, it does happen on all NNs sooner or later as far as I can tell. As I mentioned, I raised the "max collision events" from 32 to 256 because I read that some people recommended doing so. Was this perhaps a mistake? Otherwise, I use fp 16, 20 million for NNCacheSize, 3.3 for CPuct, and 10,000 for CPuctBase. Any of this seem suspicious?

This is the best configuration I have found on my system for straight games. But not back and forth analysis.
You can try it and see if it helps. But it may not if your are rocking the analysis. It may need more testing for your usage in the Chessbase GUI.
Make sure you are using the latest version of Lc0. v21.2. Correction Hash size should be set to 0

best setting for straight games.jpg

OK, I tried most of your settings (I keep threads to 2 to reserve the other 6 for Komodo). I still get the problem, even without takebacks, but perhaps it's because I'm entering the moves of the opening rather quickly in infinite mode, and perhaps it gets confused switching sides so quickly. I don't get the problem when I just start the engine from the position. A couple of settings questions:
1. Do you get a noticeable speedup from 4 threads rather than 2 (or 3)?
2. Is there a reason you keep Cache under 10 million (9,900,000 sounds like an effort to avoid exceeding 10 million)?
3. I don't see any place on my settings page for Hash Size, I am using 21.2, maybe it's due to the specific GUI?
4. Do you change settings based on time control? Perhaps blitz and standard games require rather different settings?
5. Were these settings recommended to you by someone on the Lc0 team, or did you just get them by trial and error?

Thanks.

w

1. I do get a measurable increase in running 4 threads, but not much.
2. No, you can set Cache size to any amount you have memory to support. I found as long it support your longest think times it is ok.
3. I can set my hash size in my gui. I set it to zero.
4. I set my configuration to the longest time controls. It seems to work on all time controls I test Lc0. The Setting I showed in the photo Is the setting I test at in all time controls.
5. Trial and error. This is the best setting I have found, but does not mean they are the best setting. Someone could have found better setting. But this setting is better then any A/B engine on my system.

With such different hardware configurations and the nature of Lc0. I don't know if there is a universal setting that works the best on every system.

The biggest margin of error I have found testing Lc0 is the NN them selves. Some will perform well and be dominant, and the next versrion will be 25 elo worse. But overall Lc0 is the best tested engine on my system. Only Stockfish with the up to date version is able to challenge Lc0.

[lkaufman]

Thanks. For me any difference in strength between SF and Lc0 is insignificant, what matters is that they could not be more different in how they play chess. Lc0 plays blitz like a super-GM might play correspondence chess (unaided), whereas SF plays very unhuman chess. After a few seconds, Lc0 scores change only very slightly in most positions. Stockfish, on the other hand, acts like a person with bipolar illness; even after a minute or more, the eval will suddenly swoop up or down by large amounts, with far more frequent move changes than with Lc0. For analysis, this makes Lc0 much more useful, as you never know when to stop analyzing with SF, and no two people will reach the same conclusions. But on the other hand, I do rather suspect that Lc0 just can't benefit much from long time limits. Have you done runs where you ran the same two versions of Stockfish and Lc0 against each other at different time limits? If so, have you noticed a consistent pattern? I would expect that SF would always do better at longer time limits, even if it still loses the match. That might mean that there is some time limit beyond which SF would win, you just haven't tested at such a long TC yet. Komodo MCTS acts somewhere in between Lc0 and SF, much less score fluctuation than SF, much more than Lc0.

[Werewolf]

Thanks. For me any difference in strength between SF and Lc0 is insignificant, what matters is that they could not be more different in how they play chess. Lc0 plays blitz like a super-GM might play correspondence chess (unaided), whereas SF plays very unhuman chess. After a few seconds, Lc0 scores change only very slightly in most positions. Stockfish, on the other hand, acts like a person with bipolar illness; even after a minute or more, the eval will suddenly swoop up or down by large amounts, with far more frequent move changes than with Lc0. For analysis, this makes Lc0 much more useful, as you never know when to stop analyzing with SF, and no two people will reach the same conclusions. But on the other hand, I do rather suspect that Lc0 just can't benefit much from long time limits. Have you done runs where you ran the same two versions of Stockfish and Lc0 against each other at different time limits? If so, have you noticed a consistent pattern? I would expect that SF would always do better at longer time limits, even if it still loses the match. That might mean that there is some time limit beyond which SF would win, you just haven't tested at such a long TC yet. Komodo MCTS acts somewhere in between Lc0 and SF, much less score fluctuation than SF, much more than Lc0.

I’ve been musing this point for a while. The alternative to time is hardware. I speculated that a SF cluster, top end, with 1000 cores would be too much for Lc0. That would be like running a fast workstation with more time.

But people here were unconvinced and someone referenced a match where SF had 10x more time and it didn’t help much. Unequal time matches are certainly easier than building a cluster so I’m surprised this hasn’t been tested more.

One great advantage of alpha beta is that with a deep enough search a lot of its stupidity falls away. I’m not sure that same can be said for Lc0 in positions it is weak in.