Komodo MP status?

[MM]

His list shows we need 29 elo to catch Houdini at 10 + 10, so I agree, we need another ten elo to be number one at that level. This may not be the case with MP however, we'll have to wait and see.

Note that you need more than 30 elo improvement to catch houdini1.5 at CEGT 40/20 or at CCRL 40/40 and the gap in both lists now is 35 or 36 elo.

This is because both lists show Houdini 1.5 to be 10 to 13 elo stronger than Houdini 2.0! So we only need about 25 elo to pass Houdini 2.0 on these lists, but another ten to pass 1.5! I think this means that it is quite likely that our next release will surpass Houdini 2.0 but not 1.5 on these lists!

Do you expect to have right now 25 elo margin on Komodo 4.0 to overtake Houdini and if not how much time do you think it could take?

Regards

I think we are in the 20 to 25 range now, going up about 2 elo per week.

Excellent

I bet on a Komodo stronger than Houdini even a blitz 5'+3''. I made many tests with Komodo 4.0 and the trend is that one. Then with the new Komodo....(i talk about head to head match).

Best wishes

I think you and many others have noticed that in general Komodo does better against Houdini in direct matches than would be predicted by ratings at similar time limits. I now think I know why. Houdini uses an unusually large contempt value, probably chosen to maximize its rating on the lists. This is good strategy when playing much weaker engines, but bad strategy when playing near-equals. We could raise our ratings by also using similar large contempt, but this would hurt in direct matches with Houdini. Probably we will raise it, maybe from 5 to 10. It will still be way lower than Houdini, which appears to use 29 as default.

That's right but i think Komodo positional play, by which it seems to look for the best squares for its pieces more than some tactical shots, hurts Houdini; i remember how many times the evaluation of Houdini during these head to head matches dropped cause the positional manouevres of Komodo.

Regards

[Ron Langeveld]

His list shows we need 29 elo to catch Houdini at 10 + 10, so I agree, we need another ten elo to be number one at that level. This may not be the case with MP however, we'll have to wait and see.

Note that you need more than 30 elo improvement to catch houdini1.5 at CEGT 40/20 or at CCRL 40/40 and the gap in both lists now is 35 or 36 elo.

This is because both lists show Houdini 1.5 to be 10 to 13 elo stronger than Houdini 2.0! So we only need about 25 elo to pass Houdini 2.0 on these lists, but another ten to pass 1.5! I think this means that it is quite likely that our next release will surpass Houdini 2.0 but not 1.5 on these lists!

Do you expect to have right now 25 elo margin on Komodo 4.0 to overtake Houdini and if not how much time do you think it could take?

Regards

I think we are in the 20 to 25 range now, going up about 2 elo per week.

Excellent

I bet on a Komodo stronger than Houdini even a blitz 5'+3''. I made many tests with Komodo 4.0 and the trend is that one. Then with the new Komodo....(i talk about head to head match).

Best wishes

I think you and many others have noticed that in general Komodo does better against Houdini in direct matches than would be predicted by ratings at similar time limits. I now think I know why. Houdini uses an unusually large contempt value, probably chosen to maximize its rating on the lists. This is good strategy when playing much weaker engines, but bad strategy when playing near-equals. We could raise our ratings by also using similar large contempt, but this would hurt in direct matches with Houdini. Probably we will raise it, maybe from 5 to 10. It will still be way lower than Houdini, which appears to use 29 as default.

Is this (raising the contempt value) the difference between 4371.00 and the new beta ? Running over 2000 games now in my tests and the 19/3 version (K4393.00) is trailing Houdini by 50 and 42 elo points resp. and against Critter it is up 1 elo point (all games in 3m+3). Performance against K4371.00 from February 2nd is +4 so I guess that makes the progress since early January about +7 and not +30.

Best regards,
Ron

[Don]

His list shows we need 29 elo to catch Houdini at 10 + 10, so I agree, we need another ten elo to be number one at that level. This may not be the case with MP however, we'll have to wait and see.

Note that you need more than 30 elo improvement to catch houdini1.5 at CEGT 40/20 or at CCRL 40/40 and the gap in both lists now is 35 or 36 elo.

This is because both lists show Houdini 1.5 to be 10 to 13 elo stronger than Houdini 2.0! So we only need about 25 elo to pass Houdini 2.0 on these lists, but another ten to pass 1.5! I think this means that it is quite likely that our next release will surpass Houdini 2.0 but not 1.5 on these lists!

Do you expect to have right now 25 elo margin on Komodo 4.0 to overtake Houdini and if not how much time do you think it could take?

Regards

I think we are in the 20 to 25 range now, going up about 2 elo per week.

Excellent

I bet on a Komodo stronger than Houdini even a blitz 5'+3''. I made many tests with Komodo 4.0 and the trend is that one. Then with the new Komodo....(i talk about head to head match).

Best wishes

I think you and many others have noticed that in general Komodo does better against Houdini in direct matches than would be predicted by ratings at similar time limits. I now think I know why. Houdini uses an unusually large contempt value, probably chosen to maximize its rating on the lists. This is good strategy when playing much weaker engines, but bad strategy when playing near-equals. We could raise our ratings by also using similar large contempt, but this would hurt in direct matches with Houdini. Probably we will raise it, maybe from 5 to 10. It will still be way lower than Houdini, which appears to use 29 as default.

Is this (raising the contempt value) the difference between 4371.00 and the new beta ? Running over 2000 games now in my tests and the 19/3 version (K4393.00) is trailing Houdini by 50 and 42 elo points resp. and against Critter it is up 1 elo point (all games in 3m+3). Performance against K4371.00 from February 2nd is +4 so I guess that makes the progress since early January about +7 and not +30.

Best regards,
Ron

Ron,

Can you remind me of the match conditions? Which processor, time control, hash table size, etc. ???

Don

[Uri Blass]

His list shows we need 29 elo to catch Houdini at 10 + 10, so I agree, we need another ten elo to be number one at that level. This may not be the case with MP however, we'll have to wait and see.

Note that you need more than 30 elo improvement to catch houdini1.5 at CEGT 40/20 or at CCRL 40/40 and the gap in both lists now is 35 or 36 elo.

This is because both lists show Houdini 1.5 to be 10 to 13 elo stronger than Houdini 2.0! So we only need about 25 elo to pass Houdini 2.0 on these lists, but another ten to pass 1.5! I think this means that it is quite likely that our next release will surpass Houdini 2.0 but not 1.5 on these lists!

Do you expect to have right now 25 elo margin on Komodo 4.0 to overtake Houdini and if not how much time do you think it could take?

Regards

I think we are in the 20 to 25 range now, going up about 2 elo per week.

Excellent

I bet on a Komodo stronger than Houdini even a blitz 5'+3''. I made many tests with Komodo 4.0 and the trend is that one. Then with the new Komodo....(i talk about head to head match).

Best wishes

I think you and many others have noticed that in general Komodo does better against Houdini in direct matches than would be predicted by ratings at similar time limits. I now think I know why. Houdini uses an unusually large contempt value, probably chosen to maximize its rating on the lists. This is good strategy when playing much weaker engines, but bad strategy when playing near-equals. We could raise our ratings by also using similar large contempt, but this would hurt in direct matches with Houdini. Probably we will raise it, maybe from 5 to 10. It will still be way lower than Houdini, which appears to use 29 as default.

I wonder if the large contempt may be also the reason that
komodo scales better than houdini and it may be interesting to test houdini with no contempt against komodo with no contempt to see if komodo with no contempt also scales better than houdini with no contempt.

I guess that both do not use contempt in analysis.

[Ron Langeveld]

Don,

i920 @3,2 GHz
128 MB hash
3 minutes + 3 seconds per game ponder off

[Don]

His list shows we need 29 elo to catch Houdini at 10 + 10, so I agree, we need another ten elo to be number one at that level. This may not be the case with MP however, we'll have to wait and see.

Note that you need more than 30 elo improvement to catch houdini1.5 at CEGT 40/20 or at CCRL 40/40 and the gap in both lists now is 35 or 36 elo.

This is because both lists show Houdini 1.5 to be 10 to 13 elo stronger than Houdini 2.0! So we only need about 25 elo to pass Houdini 2.0 on these lists, but another ten to pass 1.5! I think this means that it is quite likely that our next release will surpass Houdini 2.0 but not 1.5 on these lists!

Do you expect to have right now 25 elo margin on Komodo 4.0 to overtake Houdini and if not how much time do you think it could take?

Regards

I think we are in the 20 to 25 range now, going up about 2 elo per week.

Excellent

I bet on a Komodo stronger than Houdini even a blitz 5'+3''. I made many tests with Komodo 4.0 and the trend is that one. Then with the new Komodo....(i talk about head to head match).

Best wishes

I think you and many others have noticed that in general Komodo does better against Houdini in direct matches than would be predicted by ratings at similar time limits. I now think I know why. Houdini uses an unusually large contempt value, probably chosen to maximize its rating on the lists. This is good strategy when playing much weaker engines, but bad strategy when playing near-equals. We could raise our ratings by also using similar large contempt, but this would hurt in direct matches with Houdini. Probably we will raise it, maybe from 5 to 10. It will still be way lower than Houdini, which appears to use 29 as default.

I wonder if the large contempt may be also the reason that
komodo scales better than houdini and it may be interesting to test houdini with no contempt against komodo with no contempt to see if komodo with no contempt also scales better than houdini with no contempt.

I guess that both do not use contempt in analysis.

I believe that Houdini's contempt factor has nothing to do with scalability, but I do think it's inappropriate for a strong program like Komodo. There is a cause and effect issue here - contempt is not the cause, but it can be the effect, at least in a very minor way. If one program is scaling better than obviously minor adjustments to the contempt would make sense.

I think Houdini gives up very little ELO to Komodo due to aggressive contempt, but it might be as much as 5 or 6 ELO. I don't know for sure as I do not have a formula for deriving how much you give up for being wrong. But I know that when the difference is large, it makes a huge difference.

I believe that having a contempt close to zero has really hurt us on the rating lists and one of the lists we do best on only tests against the top programs where this is not much of an issue.

We are probably going to use a system where the contempt factor is pretty large in the opening and is reduced depending on the game phase. I think this will make the program more fun for people and it's more for cosmetic purposes and usability. I don't think most people want their program to constantly be seeking draws just because there is a slightly negative score. We want to get good results but not by "gaming the system", our primary consideration is to have a program that is fun to play and people want to have.

I have been running a new study for several weeks now and have built a web page. The web page has all the details and a nice graph too:

http://komodochess.com/pub/scale.html

In this study I use 32 bit Houdini because I don't have wine64 working on the computer I am using for the study but it makes the 2 programs more comparable without having to adjust the y-axis to simulate a time handicap. I want to extend this to level 7 and add other programs.

[lkaufman]

His list shows we need 29 elo to catch Houdini at 10 + 10, so I agree, we need another ten elo to be number one at that level. This may not be the case with MP however, we'll have to wait and see.

Note that you need more than 30 elo improvement to catch houdini1.5 at CEGT 40/20 or at CCRL 40/40 and the gap in both lists now is 35 or 36 elo.

This is because both lists show Houdini 1.5 to be 10 to 13 elo stronger than Houdini 2.0! So we only need about 25 elo to pass Houdini 2.0 on these lists, but another ten to pass 1.5! I think this means that it is quite likely that our next release will surpass Houdini 2.0 but not 1.5 on these lists!

Do you expect to have right now 25 elo margin on Komodo 4.0 to overtake Houdini and if not how much time do you think it could take?

Regards

I think we are in the 20 to 25 range now, going up about 2 elo per week.

Excellent

I bet on a Komodo stronger than Houdini even a blitz 5'+3''. I made many tests with Komodo 4.0 and the trend is that one. Then with the new Komodo....(i talk about head to head match).

Best wishes

I think you and many others have noticed that in general Komodo does better against Houdini in direct matches than would be predicted by ratings at similar time limits. I now think I know why. Houdini uses an unusually large contempt value, probably chosen to maximize its rating on the lists. This is good strategy when playing much weaker engines, but bad strategy when playing near-equals. We could raise our ratings by also using similar large contempt, but this would hurt in direct matches with Houdini. Probably we will raise it, maybe from 5 to 10. It will still be way lower than Houdini, which appears to use 29 as default.

Is this (raising the contempt value) the difference between 4371.00 and the new beta ? Running over 2000 games now in my tests and the 19/3 version (K4393.00) is trailing Houdini by 50 and 42 elo points resp. and against Critter it is up 1 elo point (all games in 3m+3). Performance against K4371.00 from February 2nd is +4 so I guess that makes the progress since early January about +7 and not +30.

Best regards,
Ron

I only estimate v 4393 as +20 over Komodo 4, not +30 (the latest 4396 being perhaps a couple elo stronger). Your +7 result is the worst of many which average around +20. Bear in mind that you are comparing a non-optimized compile with the optimized release compile, or else you are comparing to a version already improved from Komodo 4. I ran 1300 games for the last two versions at 5'+3" (no ponder) against Critter 1.4, and got 52% despite the fact that these were run on an old system without SSE support, which hurts us far more than Critter, and with a non-optimized compile.

[Uri Blass]

Don,The question is if big contempt does not cause the program to scale worse.

Did you do tests with big contempt to see if Komodo with big contempt(and you can use in your testing even bigger contempt than houdini) does not scale worse than komodo with no contempt?

[lkaufman]

Don,The question is if big contempt does not cause the program to scale worse.

Did you do tests with big contempt to see if Komodo with big contempt(and you can use in your testing even bigger contempt than houdini) does not scale worse than komodo with no contempt?

We haven't tested that. Ideally Don could repeat his latest test with no contempt for either program, but unfortunately it seems that even with contempt set to "0" for Houdini it is still using 15. I think it is likely that big contempt will scale poorly, but I doubt that this effect is large enough to explain our observations.

[Don]

Don,The question is if big contempt does not cause the program to scale worse.

Did you do tests with big contempt to see if Komodo with big contempt(and you can use in your testing even bigger contempt than houdini) does not scale worse than komodo with no contempt?

I did some tests earlier with different contempt factors to see what impact they had on strength. The impact is very minor when the difference between programs is less than 100 ELO.

Contempt is not going to affect scaling at all because the contempt factor to set should be a constant based on the difference in ELO between the two programs. You don't change the contempt factor based on the time control, only the difference in ELO between two programs. So if the difference in ELO changes between levels it is because of scaling issues, not the contempt factor.

Of course there can be a slight feedback effect. If one program scales better than another then as you play with longer time controls you might need to adjust the contempt factor accordingly. But that would be a side effect of scaling, not a cause that creates scaling issues.

You should run your own test - you can isolate the issue more effectively just by running the same program against itself with different contempt factors at different levels. You already know the proper contempt factor, it should be zero. So set one to 20 or 30 and see if it creates a major scaling issue. What you will see happen is that the program with the big contempt factor will play slightly weaker than the other at all levels but the difference will not increase with depth.