Komodo vs. Larry K on chess.com

[lkaufman]

It appears that the NN approach may favor Shogi more than other games. Look at LCZero/Shogi verses Elmo here:
https://deepmind.com/blog/article/alpha ... ogi-and-go

If you are referring to the comparative scores (margin of victory), you need to discard draws when comparing chess to shogi, in which case the comparison is not so clear, depending on exactly which chess match you use as the base. But I don't think Elmo has been refined to the same level as Stockfish, so it's not a fair comparison.

[Ovyron]

That's very interesting. I wonder how the chess rating lists would look like with draws removed (some kind of chess variant where if the game ends in draw it doesn't count.)

[jp]

It appears that the NN approach may favor Shogi more than other games. Look at LCZero/Shogi verses Elmo

If you are referring to the comparative scores (margin of victory), you need to discard draws when comparing chess to shogi, in which case the comparison is not so clear, depending on exactly which chess match you use as the base. But I don't think Elmo has been refined to the same level as Stockfish, so it's not a fair comparison.

There were also complaints from the Shogi/Elmo community about how they used Elmo there, so the comparison would need to judge whether the Elmo crippling in that match was the same disadvantage as the SF8 crippling.

[Ovyron]

(except SF wasn't crippled because Alpha played under the same conditions)

[lkaufman]

(except SF wasn't crippled because Alpha played under the same conditions)

Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.

[jp]

Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.

Do you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.

When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.

When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.

[lkaufman]

Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.

Do you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.

When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.

When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.

The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third. The < 5% wins figure you cite, which I think is normal in top level correspondence chess now, shows that chess starting in the initial position with standard rules is no longer a playable game between the top engines, we have to resort to preselected openings or other variations like 960 to make it viable.

[jp]

The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third.

Does the Komodo team have a recommended "best" hash size for a given number of nodes per move?

For the SF hardware & time they used, I think a typical rule of thumb gave hash size 32GB or something like that, but others were suggesting different numbers (but all of them much more than the 1GB used, and 1GB was a very strange choice, because even the cheapest laptop nowadays comes with 4GB RAM).

[lkaufman]

The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third.

Does the Komodo team have a recommended "best" hash size for a given number of nodes per move?

For the SF hardware & time they used, I think a typical rule of thumb gave hash size 32GB or something like that, but others were suggesting different numbers (but all of them much more than the 1GB used, and 1GB was a very strange choice, because even the cheapest laptop nowadays comes with 4GB RAM).

Not exactly, but we do have guidelines in our readme for how to set Hash size. 1 GB hash for a monster machine is so obviously silly that I thought at first it was a typo, maybe they meant 1 GB per thread or core, but apparently not. I can only guess that no one knew much about this or thought it through.

[Uri Blass]

Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.

Do you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.

When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.

When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.

The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third. The < 5% wins figure you cite, which I think is normal in top level correspondence chess now, shows that chess starting in the initial position with standard rules is no longer a playable game between the top engines, we have to resort to preselected openings or other variations like 960 to make it viable.

The standard position is playable between top engines as long as there are wins.
<5% wins is still not 0% wins and it is possible to give white more time to reduce the percentage of draws.

It may be interesting to know the percentage of draws in a match with no opening book from the opening position between stockfish and lc0 when white get 10 minutes per move against 1 minutes per move by black(of course no pondering).