Komodo MCTS

[AdminX]

Now try the same match at MultiPV=4 (as some people use in analysis)

I used the maximum number of cores for the engines that my P5 processor allowed, MultiPV=5. Ponder=Off.

Did you set MultiPV = 5 for ALL engines, not just Komodo?

Just Komodo.

Your point about MPV and analysis is well taken. Komodo MCTS is superb for analysis, and is even more compelling it you don't have a multi-core monster machine.

I agree, It delivers what is advertised. I was I was stunned by the results of a small match with Shredder 13 and wonder how long till it starts touching the top 3 as well. My only complaint is wtih the MTCS endgames.

[sound67]

Did you set MultiPV = 5 for ALL engines, not just Komodo? With that setting, Komodo 13.02 MCTS is MUCH stronger than normal Komodo, and about the same strength as Stockfish 10 (also using MultiPV). If you are getting results with Komodo MCTS below normal Komodo, something is wrong.

I set the MultiPV for all engines to 5, otherwise used the default values and the Perfect2017 book for each side.

According to the CCRL list (https://ccrl.chessdom.com/ccrl/4040/), even the regular Komodo 11 is stronger than the Komodo 12 MCTS, and the makers of the program also implicitly state that for version 13 as they recommend MCTS for detecting unusual moves in analysis, but regular Komodo 13 for gameplay.

The weakness of the MCTS engine came into clear focus to me when time began to run out.

[leavenfish]

I saw a screen shot somewhere of Komodo MCTS and I know it had multiple CPU's involved...3 or something as there were 3 lines being looked at. It showed something like depth = 24/17; had to be in a Chessbase GUI.

Exactly what does that depth mean in Komodo?

It struck me because I remember Ntirlis in Aagaards THINKING INSIDE THE BOX say that a depth of 21/35 means that the principle variation is 21 ply deep but there other lines where the engine was looking 35 ply deep.

[lkaufman]

I saw a screen shot somewhere of Komodo MCTS and I know it had multiple CPU's involved...3 or something as there were 3 lines being looked at. It showed something like depth = 24/17; had to be in a Chessbase GUI.

Exactly what does that depth mean in Komodo?

It struck me because I remember Ntirlis in Aagaards THINKING INSIDE THE BOX say that a depth of 21/35 means that the principle variation is 21 ply deep but there other lines where the engine was looking 35 ply deep.

With Komodo MCTS (as also with Lc0), the first number is artificial, as there is no iteration in the normal sense. It is just a number calculated from the number of MCTS nodes generated, kind of like a substitute for depth in an A/B engine. The second number is the maximum depth reached by any MCTS node, but note that there might be another 15 plies or so of search that takes place AT that terminal MCTS node. So it is far from the maximum depth reached in the search.

[Dann Corbit]

I saw a screen shot somewhere of Komodo MCTS and I know it had multiple CPU's involved...3 or something as there were 3 lines being looked at. It showed something like depth = 24/17; had to be in a Chessbase GUI.

Exactly what does that depth mean in Komodo?

It struck me because I remember Ntirlis in Aagaards THINKING INSIDE THE BOX say that a depth of 21/35 means that the principle variation is 21 ply deep but there other lines where the engine was looking 35 ply deep.

The normal meanings are as follows:
Even depth is not interpreted the same by every engine, as (for instance) some engines do not count single reply extensions.
But, in general the first number is the number of plies completed in the alpha-beta search.
The second number is selective search depth and it can mean anything the author wants.
It could be the alpha-beta depth + qsearch length.
It could be the total distance to the deepest line examined.
I think the intention was to add any search extensions but exactly what seldepth means is not clearly spelled out.
I think also that the description above is OK

Now, MCTS and NN programs have some other meaning for depth, because the do not generally even perform an alpha beta search

[rcmaddox]

I've been reluctant to post in threads about Leela or Komodo MCTS because, frankly, I don't have a good conceptual understanding of how these engines work. I hate to publicly display my limitations, but here goes...

Last night, just for kicks, I watched Komodo 13.02 MCTS play a series of games against Houdini 3 and Deep Shredder 13. Komodo MCTS did well, modestly outscoring both engines.

I noticed that if I turn off adjudication, so that a won game must be played to checkmate, Komodo would play horrible chess in trivially won endgames. For example, in one game, Komodo ended up with K and Q vs a lone K. Komodo seemed to just randomly check the opposing king, with no plan to combine with its own king to deliver checkmate. I watched Komodo chase the opposing king around the board for maybe 15 moves - then I just stopped the game. I suppose Komodo would have eventually stumbled into checkmate but life is short.

So ... Komodo made strong moves in the middle game but (seemingly) pointless moves when it had an overwhelming advantage in the endgame.

Is this normal? I did have 5-man Syzygy enabled. All this took place on my modest laptop.

[lkaufman]

I've been reluctant to post in threads about Leela or Komodo MCTS because, frankly, I don't have a good conceptual understanding of how these engines work. I hate to publicly display my limitations, but here goes...

Last night, just for kicks, I watched Komodo 13.02 MCTS play a series of games against Houdini 3 and Deep Shredder 13. Komodo MCTS did well, modestly outscoring both engines.

I noticed that if I turn off adjudication, so that a won game must be played to checkmate, Komodo would play horrible chess in trivially won endgames. For example, in one game, Komodo ended up with K and Q vs a lone K. Komodo seemed to just randomly check the opposing king, with no plan to combine with its own king to deliver checkmate. I watched Komodo chase the opposing king around the board for maybe 15 moves - then I just stopped the game. I suppose Komodo would have eventually stumbled into checkmate but life is short.

So ... Komodo made strong moves in the middle game but (seemingly) pointless moves when it had an overwhelming advantage in the endgame.

Is this normal? I did have 5-man Syzygy enabled. All this took place on my modest laptop.

This happens all the time with Lc0 as well. They know it's a win in fifty moves or less, but until they get to the point where they have to find the right moves to win in fifty moves they play randomly. So they don't miss the win, they just appear to be "trolling". Maybe we can find a better way.

[carldaman]

The Lc0 endgame trolling is supposedly due to how its NN are trained, if my understanding is correct - (and that's always doubtful ), although one would hope that with few pieces on the board the MCTS search wouldn't have that much trouble finding a more direct win.

Now, what could be Komodo's trouble here, since it uses no NNs? Is it the MCTS algorithm that's the culprit after all?
I would sure like to have some clarity on this. Komodo is still being advertised as a great positional engine, but this endgame trolling clearly goes against assumption, as far as the MCTS version goes. Can't the syzygy bases help at all?

(I suppose one could always switch back to regular non-MCTS Komodo for endgame analysis.)

[leavenfish]

I saw a screen shot somewhere of Komodo MCTS and I know it had multiple CPU's involved...3 or something as there were 3 lines being looked at. It showed something like depth = 24/17; had to be in a Chessbase GUI.

Exactly what does that depth mean in Komodo?

It struck me because I remember Ntirlis in Aagaards THINKING INSIDE THE BOX say that a depth of 21/35 means that the principle variation is 21 ply deep but there other lines where the engine was looking 35 ply deep.

With Komodo MCTS (as also with Lc0), the first number is artificial, as there is no iteration in the normal sense. It is just a number calculated from the number of MCTS nodes generated, kind of like a substitute for depth in an A/B engine. The second number is the maximum depth reached by any MCTS node, but note that there might be another 15 plies or so of search that takes place AT that terminal MCTS node. So it is far from the maximum depth reached in the search.

Interesting indeed.

Of course 'depth search' ( or something like time used) provides a user with some comfort that an engine has chewed upon a position long enough to have some measure of 'trust' in the output....or in comparing the output vs the 'depth' of search for a traditional A/B engine.

But I can see where this could be useful in my opening/early middlegame analysis. I think I will get the Chessbase version, thanks!!

[lkaufman]

The Lc0 endgame trolling is supposedly due to how its NN are trained, if my understanding is correct - (and that's always doubtful ), although one would hope that with few pieces on the board the MCTS search wouldn't have that much trouble finding a more direct win.

Now, what could be Komodo's trouble here, since it uses no NNs? Is it the MCTS algorithm that's the culprit after all?
I would sure like to have some clarity on this. Komodo is still being advertised as a great positional engine, but this endgame trolling clearly goes against assumption, as far as the MCTS version goes. Can't the syzygy bases help at all?

(I suppose one could always switch back to regular non-MCTS Komodo for endgame analysis.)

It seems that the combination of MCTS with Syzygy is the problem. When using Syzygy, all lines that mate within 50 moves look equally good to MCTS. If you turn off Syzygy MCTS will mate much more efficiently with queen vs king. I think that Syzygy does add a couple elo points to Komodo MCTS, but it can sure make it look stupid or trollish.