Komodo vs. Larry K on chess.com

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
lkaufman
Posts: 3757
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo vs. Larry K on chess.com

Post by lkaufman » Fri Aug 30, 2019 4:35 am

Dann Corbit wrote:
Fri Aug 30, 2019 1:08 am
It appears that the NN approach may favor Shogi more than other games. Look at LCZero/Shogi verses Elmo here:
https://deepmind.com/blog/article/alpha ... ogi-and-go
If you are referring to the comparative scores (margin of victory), you need to discard draws when comparing chess to shogi, in which case the comparison is not so clear, depending on exactly which chess match you use as the base. But I don't think Elmo has been refined to the same level as Stockfish, so it's not a fair comparison.
Komodo rules!

User avatar
Ovyron
Posts: 2829
Joined: Tue Jul 03, 2007 2:30 am

Re: Komodo vs. Larry K on chess.com

Post by Ovyron » Fri Aug 30, 2019 4:36 am

That's very interesting. I wonder how the chess rating lists would look like with draws removed (some kind of chess variant where if the game ends in draw it doesn't count.)
Great spirits have always encountered violent opposition from mediocre minds.

jp
Posts: 840
Joined: Mon Apr 23, 2018 5:54 am

Re: Komodo vs. Larry K on chess.com

Post by jp » Fri Aug 30, 2019 11:48 am

lkaufman wrote:
Fri Aug 30, 2019 4:35 am
Dann Corbit wrote:
Fri Aug 30, 2019 1:08 am
It appears that the NN approach may favor Shogi more than other games. Look at LCZero/Shogi verses Elmo
If you are referring to the comparative scores (margin of victory), you need to discard draws when comparing chess to shogi, in which case the comparison is not so clear, depending on exactly which chess match you use as the base. But I don't think Elmo has been refined to the same level as Stockfish, so it's not a fair comparison.
There were also complaints from the Shogi/Elmo community about how they used Elmo there, so the comparison would need to judge whether the Elmo crippling in that match was the same disadvantage as the SF8 crippling.

User avatar
Ovyron
Posts: 2829
Joined: Tue Jul 03, 2007 2:30 am

Re: Komodo vs. Larry K on chess.com

Post by Ovyron » Sat Aug 31, 2019 3:44 am

(except SF wasn't crippled because Alpha played under the same conditions)
Great spirits have always encountered violent opposition from mediocre minds.

lkaufman
Posts: 3757
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo vs. Larry K on chess.com

Post by lkaufman » Sat Aug 31, 2019 5:29 am

Ovyron wrote:
Sat Aug 31, 2019 3:44 am
(except SF wasn't crippled because Alpha played under the same conditions)
Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
Komodo rules!

jp
Posts: 840
Joined: Mon Apr 23, 2018 5:54 am

Re: Komodo vs. Larry K on chess.com

Post by jp » Sat Aug 31, 2019 12:05 pm

lkaufman wrote:
Sat Aug 31, 2019 5:29 am
Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
Do you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.

When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.

When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.

lkaufman
Posts: 3757
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo vs. Larry K on chess.com

Post by lkaufman » Sat Aug 31, 2019 3:51 pm

jp wrote:
Sat Aug 31, 2019 12:05 pm
lkaufman wrote:
Sat Aug 31, 2019 5:29 am
Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
Do you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.

When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.

When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.
The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third. The < 5% wins figure you cite, which I think is normal in top level correspondence chess now, shows that chess starting in the initial position with standard rules is no longer a playable game between the top engines, we have to resort to preselected openings or other variations like 960 to make it viable.
Komodo rules!

jp
Posts: 840
Joined: Mon Apr 23, 2018 5:54 am

Re: Komodo vs. Larry K on chess.com

Post by jp » Sat Aug 31, 2019 3:56 pm

lkaufman wrote:
Sat Aug 31, 2019 3:51 pm
The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third.
Does the Komodo team have a recommended "best" hash size for a given number of nodes per move?

For the SF hardware & time they used, I think a typical rule of thumb gave hash size 32GB or something like that, but others were suggesting different numbers (but all of them much more than the 1GB used, and 1GB was a very strange choice, because even the cheapest laptop nowadays comes with 4GB RAM).

lkaufman
Posts: 3757
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo vs. Larry K on chess.com

Post by lkaufman » Sat Aug 31, 2019 6:21 pm

jp wrote:
Sat Aug 31, 2019 3:56 pm
lkaufman wrote:
Sat Aug 31, 2019 3:51 pm
The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third.
Does the Komodo team have a recommended "best" hash size for a given number of nodes per move?

For the SF hardware & time they used, I think a typical rule of thumb gave hash size 32GB or something like that, but others were suggesting different numbers (but all of them much more than the 1GB used, and 1GB was a very strange choice, because even the cheapest laptop nowadays comes with 4GB RAM).
Not exactly, but we do have guidelines in our readme for how to set Hash size. 1 GB hash for a monster machine is so obviously silly that I thought at first it was a typo, maybe they meant 1 GB per thread or core, but apparently not. I can only guess that no one knew much about this or thought it through.
Komodo rules!

Uri Blass
Posts: 8609
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Komodo vs. Larry K on chess.com

Post by Uri Blass » Sat Aug 31, 2019 8:52 pm

lkaufman wrote:
Sat Aug 31, 2019 3:51 pm
jp wrote:
Sat Aug 31, 2019 12:05 pm
lkaufman wrote:
Sat Aug 31, 2019 5:29 am
Not really the same, because Alpha effectively had its own opening book, so for fairness either SF should have been allowed to use its own book ("Brainfish") or random book openings should have been selected. There were other lesser issues. To be fair, this was addressed a year later, when the results with prefed openings weren't so one-sided, though still good for Alpha.
Do you think the tiny hash size, for example, is a lesser issue? Some, possibly on this forum too, have run SF vs itself at different hash sizes. I recall it being a significant elo difference. I assume that is also true for Komodo, Houdini, etc., though I don't know if there's agreement on exactly the optimal hash size for given nodes per move.

When the reviewers made them partially address issues later, that was only for SF, not for Elmo, I believe, so that affects the comparison of NN for shogi vs. chess. There were many issues re. how they used Elmo, but these were not written about in English, and there is little overlap or communication between the chess and shogi (computer or non-computer) communities, so I think those issues were largely overlooked.

When they used SF8+Cerebellum book against A0, it was >90% draws, < 5% wins. That's basically just a drawn result.
The tiny hash size was probably the second most important issue, after the opening book issue. Fixed time per move was perhaps third. The < 5% wins figure you cite, which I think is normal in top level correspondence chess now, shows that chess starting in the initial position with standard rules is no longer a playable game between the top engines, we have to resort to preselected openings or other variations like 960 to make it viable.
The standard position is playable between top engines as long as there are wins.
<5% wins is still not 0% wins and it is possible to give white more time to reduce the percentage of draws.

It may be interesting to know the percentage of draws in a match with no opening book from the opening position between stockfish and lc0 when white get 10 minutes per move against 1 minutes per move by black(of course no pondering).

Post Reply