Cornfed wrote: ↑Sat May 23, 2020 2:39 am
"Well now we know, that the MCTS version is the best with odds"
But, why?
It is better because it does not assume "perfect" play by the opponent, but chooses its move based on what it thinks will score best against good but imperfect play, just like Alpha Zero and Lc0 do, but without the Neural Network. Standard Komodo may avoid moves for reasons that no human would ever imagine.
How does the MCTS version decide what move is most likely to cause human problems, even though it is not objectively the best?
Also does MCTS version score better than regular version against a human in non handicapped chess and if the advantage of MCTS version against humans is bigger in handicapped chess than regular chess why is this ?
MCTS tries different moves for both sides and sums the statistics, it doesn't assume it "knows" what the opponent will play, only how likely various moves are to be played.
Your second question has no good answer, because no human has any significant chance to score without a handicap against any top engine, unless when he has White the opening book ends in a very drawish position or if the engine isn't set to avoid draws when playing Black. Of course you can handicap by huge time odds. In that case you can't use the MCTS version, it doesn't work properly at less than around a second per move on on thread. I suppose in theory we could have Magnus Carlsen with 3 hours + 1 minute play Komodo with 3 minutes + 1 second, take White every game and draw odds, and see whether MCTS or standard Komodo scores better, but who will finance this? I don't know whether it's necessary to be tricky to avoid draws as Black or just to play good moves that avoid simplification or blockades or repetitions. If the latter is true, then standard Komodo would do better, if not MCTS might do better. With material handicaps it doesn't work to just prolong the game, you need to set problems.
Cornfed wrote: ↑Sat May 23, 2020 2:39 am
"Well now we know, that the MCTS version is the best with odds"
But, why?
It is better because it does not assume "perfect" play by the opponent, but chooses its move based on what it thinks will score best against good but imperfect play, just like Alpha Zero and Lc0 do, but without the Neural Network. Standard Komodo may avoid moves for reasons that no human would ever imagine.
How does the MCTS version decide what move is most likely to cause human problems, even though it is not objectively the best?
Also does MCTS version score better than regular version against a human in non handicapped chess and if the advantage of MCTS version against humans is bigger in handicapped chess than regular chess why is this ?
MCTS tries different moves for both sides and sums the statistics, it doesn't assume it "knows" what the opponent will play, only how likely various moves are to be played.
Your second question has no good answer, because no human has any significant chance to score without a handicap against any top engine, unless when he has White the opening book ends in a very drawish position or if the engine isn't set to avoid draws when playing Black. Of course you can handicap by huge time odds. In that case you can't use the MCTS version, it doesn't work properly at less than around a second per move on on thread. I suppose in theory we could have Magnus Carlsen with 3 hours + 1 minute play Komodo with 3 minutes + 1 second, take White every game and draw odds, and see whether MCTS or standard Komodo scores better, but who will finance this? I don't know whether it's necessary to be tricky to avoid draws as Black or just to play good moves that avoid simplification or blockades or repetitions. If the latter is true, then standard Komodo would do better, if not MCTS might do better. With material handicaps it doesn't work to just prolong the game, you need to set problems.
and why do you think Humans find MCTS version harder to play in handicapped chess than regular Komodo if MCTS version is objectively not as good ?
Cornfed wrote: ↑Sat May 23, 2020 2:39 am
"Well now we know, that the MCTS version is the best with odds"
But, why?
It is better because it does not assume "perfect" play by the opponent, but chooses its move based on what it thinks will score best against good but imperfect play, just like Alpha Zero and Lc0 do, but without the Neural Network. Standard Komodo may avoid moves for reasons that no human would ever imagine.
How does the MCTS version decide what move is most likely to cause human problems, even though it is not objectively the best?
Also does MCTS version score better than regular version against a human in non handicapped chess and if the advantage of MCTS version against humans is bigger in handicapped chess than regular chess why is this ?
MCTS tries different moves for both sides and sums the statistics, it doesn't assume it "knows" what the opponent will play, only how likely various moves are to be played.
Your second question has no good answer, because no human has any significant chance to score without a handicap against any top engine, unless when he has White the opening book ends in a very drawish position or if the engine isn't set to avoid draws when playing Black. Of course you can handicap by huge time odds. In that case you can't use the MCTS version, it doesn't work properly at less than around a second per move on on thread. I suppose in theory we could have Magnus Carlsen with 3 hours + 1 minute play Komodo with 3 minutes + 1 second, take White every game and draw odds, and see whether MCTS or standard Komodo scores better, but who will finance this? I don't know whether it's necessary to be tricky to avoid draws as Black or just to play good moves that avoid simplification or blockades or repetitions. If the latter is true, then standard Komodo would do better, if not MCTS might do better. With material handicaps it doesn't work to just prolong the game, you need to set problems.
and why do you think Humans find MCTS version harder to play in handicapped chess than regular Komodo if MCTS version is objectively not as good ?
Standard Komodo (or Stockfish) assumes that the opponent will always play what it considers to be the best move, and so in a losing position it just tries to delay the loss. But MCTS (Komodo or NN) allows for the possibility that the opponent will not play what it thinks is best, you might say it is more humble and doesn't pretend to know everything. So it maximizes practical chances rather than the chances of beating itself. It will prefer a 5% chance of winning or drawing to a sure but very delayed loss.
Cornfed wrote: ↑Sat May 23, 2020 2:39 am
"Well now we know, that the MCTS version is the best with odds"
But, why?
It is better because it does not assume "perfect" play by the opponent, but chooses its move based on what it thinks will score best against good but imperfect play, just like Alpha Zero and Lc0 do, but without the Neural Network. Standard Komodo may avoid moves for reasons that no human would ever imagine.
How does the MCTS version decide what move is most likely to cause human problems, even though it is not objectively the best?
Also does MCTS version score better than regular version against a human in non handicapped chess and if the advantage of MCTS version against humans is bigger in handicapped chess than regular chess why is this ?
MCTS tries different moves for both sides and sums the statistics, it doesn't assume it "knows" what the opponent will play, only how likely various moves are to be played.
Your second question has no good answer, because no human has any significant chance to score without a handicap against any top engine, unless when he has White the opening book ends in a very drawish position or if the engine isn't set to avoid draws when playing Black. Of course you can handicap by huge time odds. In that case you can't use the MCTS version, it doesn't work properly at less than around a second per move on on thread. I suppose in theory we could have Magnus Carlsen with 3 hours + 1 minute play Komodo with 3 minutes + 1 second, take White every game and draw odds, and see whether MCTS or standard Komodo scores better, but who will finance this? I don't know whether it's necessary to be tricky to avoid draws as Black or just to play good moves that avoid simplification or blockades or repetitions. If the latter is true, then standard Komodo would do better, if not MCTS might do better. With material handicaps it doesn't work to just prolong the game, you need to set problems.
and why do you think Humans find MCTS version harder to play in handicapped chess than regular Komodo if MCTS version is objectively not as good ?
Standard Komodo (or Stockfish) assumes that the opponent will always play what it considers to be the best move, and so in a losing position it just tries to delay the loss. But MCTS (Komodo or NN) allows for the possibility that the opponent will not play what it thinks is best, you might say it is more humble and doesn't pretend to know everything. So it maximizes practical chances rather than the chances of beating itself. It will prefer a 5% chance of winning or drawing to a sure but very delayed loss.
Just saw you have answered that question already. My apologies
Komodo has done well in this match considering it was the underdog. Can you sum up on how it played and where you want to take this from here in terms of the next match ?
duncan wrote: ↑Mon Jun 01, 2020 11:02 am
Komodo has done well in this match considering it was the underdog. Can you sum up on how it played and where you want to take this from here in terms of the next match ?
Clearly the main takeaway was that MCTS did dramatically better than Standard Komodo under these conditions. MCTS lost only two games out of 12. In the first of those, it seemed way too willing to allow exchanges. We have already made a change to Komodo based on that game that would address this problem; we already had code to deal with it, but the weight was too low, we increased it by 50%. In the second loss Komodo played for a blockade, but that was unrealistic, although it did require some skill by Lenderman to show why. Both of these problems could have been helped by setting Contempt higher; I optimized it based on games vs. crippled Komodo, but that may have been misleading.
The other general lesson from the match is that with such large handicaps, Komodo will always be playing for a draw. In theory Contempt could be set high enough (if we raised the max) to avoid this, but then it would just play badly and lose. So I think that for future matches, especially at full knight odds, it should be explicitly stated that the odds are Armageddon knight odds, meaning that the odds-giver wins draws. Even a fairly weak player can draw at knight odds with top engines as they will co-operate to head towards some drawn endgame. I think that our target should be to defeat a grandmaster (over 2500 FIDE) in a knight odds Armageddon match at 15' + 10". Although the Smerdon match makes that look impossible (no draws, so still 5 to 1 score with Armageddon rules), I believe it will happen, though not without a major breakthru. Most likely this will involve Neural Networks; we are working hard with chess.com on a Komodo NN, but no telling when it will be ready for Prime Time.
For the near time the most likely event would be one that attempts to match the high but crippled Komodo levels with grandmasters in standard chess to determine accurate ratings for them.
Would a NN Komodo play better than Komodo MTC at Rook Handicap ? It seems to me than lc0 is good at knight handicap but less at rook or queen handicap.
JJJ wrote: ↑Mon Jun 01, 2020 7:01 pm
Would a NN Komodo play better than Komodo MTC at Rook Handicap ? It seems to me than lc0 is good at knight handicap but less at rook or queen handicap.
Well, most versions of Lc0 are awful at knight handicap (and worse at rook!), except for 11248/11258; they just blunder more material. Basically they don't show much worse eval for being down two pieces or a queen than being down one minor piece. If you know of any more recent Lc0 version that can beat strong human players at knight odds, please let us know which one. But there is nothing that says NNs have to be bad at handicap play, it just requires paying attention to playing well in 99% decided situations. We really don't know what version of Komodo NN might end up a winner, but I think it's likely to be good at this if it is some sort of hybrid, as I expect.