Komodo 5 release now available!

[geots]

K5 doing well in my tests so far.

@ 1+1 , 198 games so far , K5 is ahead of Houdini 2.0c :

+69 , -46 , =83


running 300 games

Tom

I'll bet you have an Intel machine. Am I right? The present compile may well be stronger than H2 on Intel, weaker on AMD.

Larry, I think there may be a chance that K5 has caught Houdini 2.0c. But 1+1 is going to tell you absolutely nothing you can hang a hat on. But I am aware with you being 1000 times smarter than me with this stuff, you already know that.

My main machine is the same as Don's- I believe. Intel i5-2320 CPU @3.00GHz with 4 true cores and no hyperthreading. I am starting 2 matches at the same time right now:

Match 1- Houdini 2.0c v Komodo 5 w/100 games at 5m+3s

Match 2- When I tested for CCRL, we all benchmarked our computers to get them as close as possible. So if I was running this match for them at 40/40, the machine is fast enough that after benchmarking it, I would run 40/40 at 40/21. So this 2nd match will be:

Houdini 2.0c v Komodo 5 w/50 games at 40/21

Nothing written in stone with the results, but it's a darn good start at both blitz and slightly higher than intermediate controls.

**I think you will understand that it is important for me to hear from either you or Don that it is ok to publish/post regular updates from both in "Tournaments and Matches" section here. At least hearing that is important to me. NOT THAT YOU WOULD WANT ME TO HIDE BAD RESULTS- if there were any- but you might prefer FOR OTHER REASONS that I just post final results. I would feel better if it made a difference that I was told.


Best,

george

PS: It's going to be a hell of a lot closer in both than most think.

Since this is a public release and not a private beta version, you don't need our permission to post results. But any I'm fine with it, especially as you have an Intel machine -- any results for AMD machines might turn out to be moot if we issue a new compile to address the AMD problem.
I would add that the 1'+ 1" results can confirm the AMD problem, if someone else runs a similar test on an AMD machine. In general, we do recommend that private testers use the same number of minutes for base time as seconds for increment, though of course other time limits are also quite valid, except for very short sudden death time controls (like 1' or less) where time forfeits and near-forfeits play a huge role.

Look Larry, if you have to put out a new release for AMD reasons, are all the people running Intel supposed to just keep what we have now, or is it possible we would benefit from the new release in some kind of minor way? If so, I can wait to start until this is settled.


george

If there is a new release for AMD, it will probably be nearly identical to the present one for Intel machines, and we would tell everyone to treat them as identical on Intel. So go ahead. Of course anything is possible in life, maybe we'll discover that it suddenly runs twice as fast (and maybe it will snow tomorrow in midsummer!).

My wife be washing my insulated underwear tonight. Thanks.

gts

[Don]

A kind soul gave me access to an AMD windows machine and I ran this same test. As it turns out Komodo runs a little BETTER on the AMD machine. Of course this is just one machine and 1 position - not always representative. I ran the test also on my intel machine booted into windows. Anyway, we have a lot more data from various sources that say there is no AMD issue.

Don

Here are the IPON final results, prime time

Komodo 5

Komodo 5 - Houdini 2.0 STD (3028) 70.5 - 79.5 47.00% Perf=3008
Komodo 5 - Critter 1.6a (2973) 85.5 - 64.5 57.00% Perf=3021
Komodo 5 - Stockfish 2.2.2 JA (2966) 82.5 - 67.5 55.00% Perf=3000
Komodo 5 - Deep Rybka 4.1 (2956) 81.5 - 68.5 54.33% Perf=2986
Komodo 5 - Naum 4.2 (2834) 115.5 - 34.5 77.00% Perf=3043
Komodo 5 - Deep Shredder 12 (2800) 115.5 - 34.5 77.00% Perf=3009
Komodo 5 - Gull 1.2 (2796) 122.0 - 28.0 81.33% Perf=3051
Komodo 5 - Hannibal 1.2 (2792) 110.5 - 39.5 73.67% Perf=2970
Komodo 5 - Deep Sjeng c't 2010 32b (2791) 121.5 - 28.5 81.00% Perf=3042
Komodo 5 - Spike 1.4 32b (2782) 125.5 - 24.5 83.67% Perf=3065
Komodo 5 - spark-1.0 (2764) 127.0 - 23.0 84.67% Perf=3060
Komodo 5 - Protector 1.4.0 (2755) 126.0 - 24.0 84.00% Perf=3043
Komodo 5 - Deep Junior 13.3 (2744) 126.0 - 24.0 84.00% Perf=3032
Komodo 5 - HIARCS 13.2 MP 32b (2743) 127.0 - 23.0 84.67% Perf=3039
Komodo 5 - Quazar 0.4 (2732) 124.0 - 26.0 82.67% Perf=3003
Komodo 5 - Zappa Mexico II (2699) 130.5 - 19.5 87.00% Perf=3029
Komodo 5 - MinkoChess 1.3 (2691) 124.0 - 26.0 82.67% Perf=2962
1915.0 - 635.0 75.10% Perf=3005


2550 out of 2550 games played
Level: 5 Minutes/Game + 3 Seconds/Move


Assuming an AMD optimized executable makes you win circa 20 ELO over that one, you could pass Houdini 2c on this list.
Inasmuch as Houdart has an AMD optimized compile(?), it is worth another attempt, no?

It's not clear to us whether Komodo really suffers or not, but I don't think it is as much as 20 ELO. A test I would like to see someone run is this:

Run a time test on Intel using any program such as Houdini or Critter - and compare it directly to Komodo, perhaps running the opening position to depth 18 or something unless you have a more comprehesive timing test.

The run the same test on an AMD machine and get the ratio of it. See if they are comparable. Could someone do that on AMD? Here is what I suggest - since I have only Houdini 1.5:

To be consisent, open up a command prompt in Houdini 1.5 and do this:

setoption name Threads value 1
position startpos
do depth 18

Note the final line with a time.

Then do it with Komodo 5. I will do this same test on my Intel machine and we can compare ratio. Maybe there is a problem, many there is not.

Don

[lkaufman]

Isn't it possible the AMD platform used by IPON needs a different Komodo 5 compile ? I may be mistaken but wasn't this an issue with earlier versions as well ?

At the moment, IPON is showing +23 elo while our own testing network is showing +36 (at a somewhat faster time control). Ingo indicated that our AMD compile could be about 10% slower than it "should" be, which would be another ten elo. So there is no mystery anymore. The only question is whether the AMD problem justifies a new compile.

From talking to Ray, who is testing Komodo 5 on an AMD Phenom II for CCRL 40/40, he seems to think that a different compile would be pretty pointless.
It is leading Critter, Houdini, Stockfish and Rybka, so there can't be too much wrong with it.
A 5 Elo gain would be lost in the error margins anyway.

At the moment it now looks like you are right. I think it was mostly just sample error that made us think AMD compile was bad.

[MM]

If i am wrong i will apologize with Don and larry but i think this version has (if) a very little improvement compared to Komodo 4, i would say around 10 elo.

If it is so it is a very sad day for all computer chess lovers and the triumph of Robert Houdart.

Best Regards

We have MANY tester who have verified that Komodo 5 is much stronger than Komodo 4. This is based on thousands of games - and our own testing is showing over 30 ELO at 2+2 on very fast hardware - roughly equivalent to the IPON time control.

So I trust our 30,000 game sample much more than the 700 game sample - although I admit this is disappointing so far.

4448.00 and 4452.00 are very close to Komodo 5 -hardly any differences and Komodo 5 is just a better compiled version of these.

So let's wait and see what happens.

Code: Select all

Rank Name                       Elo      +      -    games   score   oppo.   draws 
   1 Komodo 5 64bit           3042.1   10.8   10.8    3097   56.8%  2995.2   43.8% 
   2 Komodo 4452.00 64 bit    3031.2    5.8    5.8   10793   55.2%  2995.4   44.4% 
   3 Komodo 4448.00 64 bit    3030.9   12.7   12.7    2251   56.5%  2985.8   43.4% 
   4 Houdini 1.5a x64         3025.9    7.3    7.3    6935   50.0%  3025.9   41.2% 
   5 Komodo 4445.00 64 bit    3024.3   10.4   10.4    3392   54.8%  2991.5   43.8% 
   6 Komodo 4 SSE4            3006.4   10.5   10.5    3381   52.1%  2991.7   40.2% 
   7 Critter 1.4 64-bit SSE4  3000.0    6.4    6.4    8696   46.3%  3025.4   45.2% 
   8 Stockfish 2.2.2 JA       2960.8    6.5    6.5    8677   40.5%  3026.7   44.0% 

Hi, i am sorry, my prediction of ''around 10 elo'' of improvement is probably wrong althought now i would wait to see cegt and ccrl. If i read correctly the result of ipon chess it is about 23 elo.

But there are some things that don't convince me.

1. I expected much more than +23 after almost a year from Komodo 4.
2. The results of tests of Komodo's team seem to show a much bigger improvement, especially against some engines (e.g. Stockfish, but in this case, at ipon chess, the gap is much smaller).
3. Some sites report bad results of Komodo 5 against Houdini 2 and against Houdini 1.5.





http://www.chess2u.com/t6106-komodo-5-tests

http://www.chess2u.com/t6105-komodo-5-v ... 1500-games

Believe me, i am in total good faith but i don't understand these results against Houdini when the internal tests of Komodo's team showed clearly another rating list.

Has someone an explaination for that?

And, most important, if Houdini continues to be clearly the number one and probably will enlarge the gap in september with Houdini 3, what is the plan of Komodo.

Personally i would have considered ''good'' an improvement of Komodo of, at least, 40 elo. So i am bit disappointed, for now.

Thank you.

Best Regards

[Don]

If i am wrong i will apologize with Don and larry but i think this version has (if) a very little improvement compared to Komodo 4, i would say around 10 elo.

If it is so it is a very sad day for all computer chess lovers and the triumph of Robert Houdart.

Best Regards

We have MANY tester who have verified that Komodo 5 is much stronger than Komodo 4. This is based on thousands of games - and our own testing is showing over 30 ELO at 2+2 on very fast hardware - roughly equivalent to the IPON time control.

So I trust our 30,000 game sample much more than the 700 game sample - although I admit this is disappointing so far.

4448.00 and 4452.00 are very close to Komodo 5 -hardly any differences and Komodo 5 is just a better compiled version of these.

So let's wait and see what happens.

Code: Select all

Rank Name                       Elo      +      -    games   score   oppo.   draws 
   1 Komodo 5 64bit           3042.1   10.8   10.8    3097   56.8%  2995.2   43.8% 
   2 Komodo 4452.00 64 bit    3031.2    5.8    5.8   10793   55.2%  2995.4   44.4% 
   3 Komodo 4448.00 64 bit    3030.9   12.7   12.7    2251   56.5%  2985.8   43.4% 
   4 Houdini 1.5a x64         3025.9    7.3    7.3    6935   50.0%  3025.9   41.2% 
   5 Komodo 4445.00 64 bit    3024.3   10.4   10.4    3392   54.8%  2991.5   43.8% 
   6 Komodo 4 SSE4            3006.4   10.5   10.5    3381   52.1%  2991.7   40.2% 
   7 Critter 1.4 64-bit SSE4  3000.0    6.4    6.4    8696   46.3%  3025.4   45.2% 
   8 Stockfish 2.2.2 JA       2960.8    6.5    6.5    8677   40.5%  3026.7   44.0% 

Hi, i am sorry, my prediction of ''around 10 elo'' of improvement is probably wrong althought now i would wait to see cegt and ccrl. If i read correctly the result of ipon chess it is about 23 elo.

But there are some things that don't convince me.

1. I expected much more than +23 after almost a year from Komodo 4.

You must be using fuzzy math. We release Komodo 4 in late Dec early Jan. It's a little more than 6 months.

2. The results of tests of Komodo's team seem to show a much bigger improvement, especially against some engines (e.g. Stockfish, but in this case, at ipon chess, the gap is much smaller).
3. Some sites report bad results of Komodo 5 against Houdini 2 and against Houdini 1.5.


http://www.chess2u.com/t6106-komodo-5-tests

http://www.chess2u.com/t6105-komodo-5-v ... 1500-games

Believe me, i am in total good faith but i don't understand these results against Houdini when the internal tests of Komodo's team showed clearly another rating list.

Has someone an explaination for that?

I think we simply need to see a lot of results under a lot of different conditions before drawing any conclusions. Larry and I should know better as we see this all the time with our testing, really good or really bad results after a few hundred games then a complete change. It's call statistical noise, sample error, whatever.

So let's just want before drawing any conclusions. Take a lesson from what happened on IPON, a terrible start that finished better. The IPON sample size is really way too low too as it's about 10 ELO in either direction for any 2 programs you are comparing. Houdini could be over or under rated and so could Komodo. Based on our results I would guess (and it's only a guess) that Houdini is slightly over-rated and Komodo is slightly under-rated. But that is pure speculation without the samples.

And, most important, if Houdini continues to be clearly the number one and probably will enlarge the gap in september with Houdini 3, what is the plan of Komodo.

We want to be number one but it seems others are more obsessed with it that we are. We feel that if we keep improving Komodo it will end up there - either sooner or later. We cannot promise that because we don't know.

I'm pretty sure there are some exciting improvements ahead, especially with code improvements which I have never put a huge amount of attention into. Komodo has the slowest nodes per second of any program in it's class and when I focus my attention on that there is a good chance it will be a pretty major thing. We also have a long list of ideas that we will be pursuing and some of them will pay off for us.

Peronally i would have considered ''good'' an improvement of Komodo of, at least, 40 elo. So i am bit disappointed, for now.

Don't forget that when you have a program as strong as Komodo you don't just add 40 ELO like it was nothing. It's only been 6 months and there is only reasonable data from one list now and our distributed testing (on other peoples machines) show about 40 ELO with FAR more games than IPON. We will likely do much better on other lists - we shall see.

Thank you.

Best Regards

[Lion]

And i would rather prefer that we see the end of the wait for the MP version earlier than see the already limited time Don has spent on an AMD compile that would bring these same owners anyway less ELO than an finally to much awaited MP.

Thanks

[carldaman]

What seems to be left out of the equation is the much awaited (and still awaited) Komodo MP. With all its near competitors being MP, I think Komodo could benefit a great deal from a MP version and perhaps clearly overtake Houdini 2 in that case.

I understand there are delays/difficulties with its implementation, but many people hoped or believed that Komodo *4* MP already would have been strong enough to catch Houdini, so there's more reason to believe that about Komodo 5 MP when its time comes.

I might also add that Houdini also benefits from having egtb support. Not sure, but has anyone quantified the value of egtb into elo points? As far as I know, many test with egtb on, again putting Komodo at a disadvantage.

These missing pieces of the puzzle could give the already very strong current Komodo that necessary edge to be first overall. Personally, I'd be happy if it were #1 at longer time controls -- I don't understand the obsession with blitz levels; heck, I think Komodo is great even without being the highest rated.

Best regards,
Carl

[carldaman]

And i would rather prefer that we see the end of the wait for the MP version earlier than see the already limited time Don has spent on an AMD compile that would bring these same owners anyway less ELO than an finally to much awaited MP.

Thanks

+1

[Laskos]

I might also add that Houdini also benefits from having egtb support. Not sure, but has anyone quantified the value of egtb into elo points? As far as I know, many test with egtb on, again putting Komodo at a disadvantage.


Best regards,
Carl

Egtb's worth is no more than 3-5 Elo points, if at all, could even harm the strength. Great for analysis, though.

Kai

[carldaman]

I might also add that Houdini also benefits from having egtb support. Not sure, but has anyone quantified the value of egtb into elo points? As far as I know, many test with egtb on, again putting Komodo at a disadvantage.


Best regards,
Carl

Egtb's worth is no more than 3-5 Elo points, if at all, could even harm the strength. Great for analysis, though.

Kai

It depends on the engine. For example, Rybka suffers a lot from frequent egtb access, slowing it down, but Houdini doesn't seem to show a performance hit, and I have to assume egtb access helps it.

CL