Komodo 8 results summary

[lkaufman]

In principle having too many similar opponents could be a problem, just as you say, but I don't think it's much of a problem for any of the lists quoted here. I think they typically include one "true" IPP at most (say Fire), plus Houdini plus a couple other engines like Critter that are quite similar to Ipp but not true clones. If 4 out of 16 or so engines are pretty similar, that does give more weight to performance against that type of engine, but since most of the strong engines other than Stockfish and Komodo are somewhat related to the IPPs, it's hard to say that doing this is wrong. It wouldn't be good if half of the engines tested were closely related, but a quarter seems acceptable to me when they are so heavily represented at the top end. I think that including opponents more than about 200 elo below the engine being rated is at least as serious a problem as insufficient diversity; striking the right balance between diversity of style and closeness of rating is difficult and a subjective matter. In our own testing we used only Stockfish and Houdini as opponents until recently, now usually only Stockfish, and doing this didn't seem to hurt our progress.

[Frank Quisinsky]

Hi Larry,

at first, thanks for your answer, perfect for me.

+1

I think too that more as 200 Elo difference (for me more around 250-300 Elo) is a problem and thus a problem in my own FCT1 work.

My work isn't perfect for that reason, YES!
I know that!!

But we can twist and turn it.
With or without clones / derivates, similar engines.

Not nice to see that from 0 to 100 20 new programs are directly 100-125 Elo stronger as the long-years development Shredder. Better as the work from Bob Hyatt, and all the other top programmers. So many Genius-People, never seen all the years before!

Curious that we try with violence to compare apples with pears and used for it complicated mathematical formulas. A typical human skill I think.

So many people try in computer chess to make the own work perfect. But we have to learn again and again ... not really possible.

Best
Frank

Short hint:
In your testing GullChess is a good opponent and Chiron. I saw much games Chiron comes in advantage. Results vs. Chiron are clear better for K8 but I am sure you can see in your work a lot of interesting things in testing vs. Chiron and of course vs. GullChess.

[Uri Blass]

Congratulations to Larry Kaufman and Mark Lefler for the remarkable gain of strength of Komodo 8. I have yet to buy it, but after these results, the thought of having it is so tempting.

I have a few questions for the Komodo team. I think it would be nice for a commercial program like Komodo to implement some non-strength related features, such as tactical mode (ala Houdini) and Chess960.

Do you have any plan to implement tactical mode and/or Chess960 into Komodo? If so, when? I just thought that it nice to have Chess960 support in Komodo, as it might improve the development, especially in the opening, as well as being a great tool for a FRC fan like me.

I know in the previous versions of Komodo (dont know about 8) have the option to turn on/off null move and LMR to improve on tactics, but it would be really nice to have a special tactical mode in Komodo.

Based on most of the rating lists available, Stockfish seems to have a slight advantage against Komodo, but lacking the ability to grind out results against inferior programs. It is the Houdini situation all over again. Perhaps Komodo has a better contempt setting for rating lists, or Stockfish need to implement one for default setting. As the TCEC is just around the corner, do you think Komodo needs a change to its contempt setting, to be able to grind out results vs Stockfish?

Thanks...

Yes, contempt helps us a bit vs. weaker engines and hurts a bit vs. SF, but the effect is pretty small. For straight up matches with SF I would set it to zero. In TCEC we leave contempt on until the semi-final or even the final, then we use zero.
Maybe we'll have 960 support in next release, no promise though. As for tactical mode, we could even add an adaptable parameter that makes Komodo as tactical as you want, and it would only take a few minutes work by Mark. But it might not be the best way to implement tactical mode. No reason we couldn't have this for next release.

I think that it may be interesting to know the advantage that you can get from contempt.

Suppose that you play against player who is 200 elo weaker(assuming contempt=0).
How much elo can you get from increasing contempt?

Same question when you play against player who is 200 elo stronger when you need negative contempt.

Can Komodo get more elo from contempt if the 200 elo advantage is because of hardware difference and not because of playing an inferior engine?

[ouachita]

Larry,
Congrats on K8 as it is clearly much stronger than K7a. My (6-core, with HT and TB, at 2m+0) test versus SF0824 has SF at +21 over 500 games and I feel sure that this delta will be less over say 5000 games.

[Leto]

Congratulations to Larry Kaufman and Mark Lefler for the remarkable gain of strength of Komodo 8. I have yet to buy it, but after these results, the thought of having it is so tempting.

I have a few questions for the Komodo team. I think it would be nice for a commercial program like Komodo to implement some non-strength related features, such as tactical mode (ala Houdini) and Chess960.

Do you have any plan to implement tactical mode and/or Chess960 into Komodo? If so, when? I just thought that it nice to have Chess960 support in Komodo, as it might improve the development, especially in the opening, as well as being a great tool for a FRC fan like me.

I know in the previous versions of Komodo (dont know about 8) have the option to turn on/off null move and LMR to improve on tactics, but it would be really nice to have a special tactical mode in Komodo.

Based on most of the rating lists available, Stockfish seems to have a slight advantage against Komodo, but lacking the ability to grind out results against inferior programs. It is the Houdini situation all over again. Perhaps Komodo has a better contempt setting for rating lists, or Stockfish need to implement one for default setting. As the TCEC is just around the corner, do you think Komodo needs a change to its contempt setting, to be able to grind out results vs Stockfish?

Thanks...

Yes, contempt helps us a bit vs. weaker engines and hurts a bit vs. SF, but the effect is pretty small. For straight up matches with SF I would set it to zero. In TCEC we leave contempt on until the semi-final or even the final, then we use zero.
Maybe we'll have 960 support in next release, no promise though. As for tactical mode, we could even add an adaptable parameter that makes Komodo as tactical as you want, and it would only take a few minutes work by Mark. But it might not be the best way to implement tactical mode. No reason we couldn't have this for next release.

I think that it may be interesting to know the advantage that you can get from contempt.

Suppose that you play against player who is 200 elo weaker(assuming contempt=0).
How much elo can you get from increasing contempt?

Same question when you play against player who is 200 elo stronger when you need negative contempt.

Can Komodo get more elo from contempt if the 200 elo advantage is because of hardware difference and not because of playing an inferior engine?

If the 200 elo advantage is from hardware difference it wouldn't be any different than playing an engine that is 200 elo weaker on equal hardware. A strong engine on hardware that is 200 elo weaker means that it doesn't have enough time to calculate enough to make strong enough moves. And an engine that is 200 elo weaker on equal hardware simply has flawed logic.

[carldaman]

Congratulations to Larry Kaufman and Mark Lefler for the remarkable gain of strength of Komodo 8. I have yet to buy it, but after these results, the thought of having it is so tempting.

I have a few questions for the Komodo team. I think it would be nice for a commercial program like Komodo to implement some non-strength related features, such as tactical mode (ala Houdini) and Chess960.

Do you have any plan to implement tactical mode and/or Chess960 into Komodo? If so, when? I just thought that it nice to have Chess960 support in Komodo, as it might improve the development, especially in the opening, as well as being a great tool for a FRC fan like me.

I know in the previous versions of Komodo (dont know about 8) have the option to turn on/off null move and LMR to improve on tactics, but it would be really nice to have a special tactical mode in Komodo.

Based on most of the rating lists available, Stockfish seems to have a slight advantage against Komodo, but lacking the ability to grind out results against inferior programs. It is the Houdini situation all over again. Perhaps Komodo has a better contempt setting for rating lists, or Stockfish need to implement one for default setting. As the TCEC is just around the corner, do you think Komodo needs a change to its contempt setting, to be able to grind out results vs Stockfish?

Thanks...

Komodo 8 retains the same UCI parameters as Komodo 7, so you can indeed turn off LMR and null move, for example. I have not yet started looking at a "tactical version". I looked into 960, but there was not enough time to add and test it before the Komodo 8 release. I have a list of things to work on, given time. But we tend to spend most of our development time on making the core program better at standard chess. But we always listen to customer requests, and they do influence what we work on.

Mark

Hi Mark,

One feature that is sorely missing from Komodo is having a learning file, or permanent hash. Komodo delivers great analysis for the most part. you'd agree, but it's a shame the user can't automatically save any of it, either for the same analysis session or later retrieval.

With all this focus on ratings and testing, one can forget the main reason many people buy a strong program, and that is analysis. Other top programs have such a feature in one form or another. Houdini has it, Rybka had it, and also freeware Critter. Even the Stockfish fork done by Jeremy Bernstein can save analysis in a learning file. I know Dann Corbit is a big fan of JB's implementation.

The great Elo gains are welcome, but those who pay for the software would like such useful features as well.

Thanks,
CL

[lkaufman]

Congratulations to Larry Kaufman and Mark Lefler for the remarkable gain of strength of Komodo 8. I have yet to buy it, but after these results, the thought of having it is so tempting.

I have a few questions for the Komodo team. I think it would be nice for a commercial program like Komodo to implement some non-strength related features, such as tactical mode (ala Houdini) and Chess960.

Do you have any plan to implement tactical mode and/or Chess960 into Komodo? If so, when? I just thought that it nice to have Chess960 support in Komodo, as it might improve the development, especially in the opening, as well as being a great tool for a FRC fan like me.

I know in the previous versions of Komodo (dont know about 8) have the option to turn on/off null move and LMR to improve on tactics, but it would be really nice to have a special tactical mode in Komodo.

Based on most of the rating lists available, Stockfish seems to have a slight advantage against Komodo, but lacking the ability to grind out results against inferior programs. It is the Houdini situation all over again. Perhaps Komodo has a better contempt setting for rating lists, or Stockfish need to implement one for default setting. As the TCEC is just around the corner, do you think Komodo needs a change to its contempt setting, to be able to grind out results vs Stockfish?

Thanks...

Yes, contempt helps us a bit vs. weaker engines and hurts a bit vs. SF, but the effect is pretty small. For straight up matches with SF I would set it to zero. In TCEC we leave contempt on until the semi-final or even the final, then we use zero.
Maybe we'll have 960 support in next release, no promise though. As for tactical mode, we could even add an adaptable parameter that makes Komodo as tactical as you want, and it would only take a few minutes work by Mark. But it might not be the best way to implement tactical mode. No reason we couldn't have this for next release.

I think that it may be interesting to know the advantage that you can get from contempt.

Suppose that you play against player who is 200 elo weaker(assuming contempt=0).
How much elo can you get from increasing contempt?

Same question when you play against player who is 200 elo stronger when you need negative contempt.

Can Komodo get more elo from contempt if the 200 elo advantage is because of hardware difference and not because of playing an inferior engine?

I think that the main effect of contempt (at least the way we do it) is so that when Komodo plays Black, it will not seek repetitions just because the book normally ends in a White plus. So the benefit should be pretty much just a question of elo difference, it shouldn't matter very much why the gap exists.

[shrapnel]

I think that the main effect of contempt (at least the way we do it) is so that when Komodo plays Black, it will not seek repetitions just because the book normally ends in a White plus. So the benefit should be pretty much just a question of elo difference, it shouldn't matter very much why the gap exists.

Hi Larry
While we are talking of Komodo playing Black, I have found a rather odd paradox in my online matches, which probably isn't very logical, but yet to be disproved, i.e., Komodo plays much better with Black with DrawScore -7 rather than with 0 !
Surprisingly, DrawScore 0 is more required when Komodo is using White, otherwise it tends to lose, at least in LTC games !
I have found this out in a series of online matches with a strong opponent who was using latest Stockfish Dev Version on a i7 like mine.
Any idea why this is happening ?
Also, Komodo seems to work very well without Book in LTC matches at least, probably because most Books have been made with the help of Houdini and Stockfish, which I suppose don't really suit Komodo's playing style and so actually end up hampering Komodo !

[lkaufman]

I think that the main effect of contempt (at least the way we do it) is so that when Komodo plays Black, it will not seek repetitions just because the book normally ends in a White plus. So the benefit should be pretty much just a question of elo difference, it shouldn't matter very much why the gap exists.

Hi Larry
While we are talking of Komodo playing Black, I have found a rather odd paradox in my online matches, which probably isn't very logical, but yet to be disproved, i.e., Komodo plays much better with Black with DrawScore -7 rather than with 0 !
Surprisingly, DrawScore 0 is more required when Komodo is using White, otherwise it tends to lose, at least in LTC games !
I have found this out in a series of online matches with a strong opponent who was using latest Stockfish Dev Version on a i7 like mine.
Any idea why this is happening ?
Also, Komodo seems to work very well without Book in LTC matches at least, probably because most Books have been made with the help of Houdini and Stockfish, which I suppose don't really suit Komodo's playing style and so actually end up hampering Komodo !

Setting drawscore to -7 has much more effect when Komodo is Black, because the opening (with or without book) usually leaves Black down by more than this, and so it will often cause Black to avoid a quick draw right out of book. With White Komodo will not often be down this much early in the game, and later if it is down that much it might mean that the opposing machine is playing better in that particular game and so a draw is much more desirable. So it makes some sense. But I would not expect it to be very noticeable.

[Vinvin]

I think that the main effect of contempt (at least the way we do it) is so that when Komodo plays Black, it will not seek repetitions just because the book normally ends in a White plus. So the benefit should be pretty much just a question of elo difference, it shouldn't matter very much why the gap exists.

Hi Larry
While we are talking of Komodo playing Black, I have found a rather odd paradox in my online matches, which probably isn't very logical, but yet to be disproved, i.e., Komodo plays much better with Black with DrawScore -7 rather than with 0 !
Surprisingly, DrawScore 0 is more required when Komodo is using White, otherwise it tends to lose, at least in LTC games !
I have found this out in a series of online matches with a strong opponent who was using latest Stockfish Dev Version on a i7 like mine.
Any idea why this is happening ?
Also, Komodo seems to work very well without Book in LTC matches at least, probably because most Books have been made with the help of Houdini and Stockfish, which I suppose don't really suit Komodo's playing style and so actually end up hampering Komodo !

Setting drawscore to -7 has much more effect when Komodo is Black, because the opening (with or without book) usually leaves Black down by more than this, and so it will often cause Black to avoid a quick draw right out of book. With White Komodo will not often be down this much early in the game, and later if it is down that much it might mean that the opposing machine is playing better in that particular game and so a draw is much more desirable. So it makes some sense. But I would not expect it to be very noticeable.

To get a more clear view, what drawscore have to be set when :
Komodo 8 is black and opponent is behind by :
50 Elo ?
100 Elo ?
150 Elo ?
200 Elo ?

Komodo 8 is white and opponent is behind by :
50 Elo ?
100 Elo ?
150 Elo ?
200 Elo ?