lkaufman wrote:Actually, I realized there is no great need to run this, because we can get the relative ratings at different time controls from the rating lists. Here is the data. Whenever both Houdini 2.0 and 1.5 appear on the same rating list in 32 bit, I averaged the ratings to get a bigger sample, since overall results for the two versions show hardly any difference (1 elo averaging the four lists cited). I also average Komodo 64 bit SSE and non-SSE when both appear on the same lists for the same reasons.
Here are the differences, Komodo 3 64 bit minus Houdini 64 bit:
40/4 (CEGT) : -35 (no CCRL data for H32 at this speed)
40/10(SWCR): -21
40/20 (CEGT): -15
40/40 (CCRL): +3
Notice the trend? It seems the point at which Komodo 3 pulls ahead appears to be at or just below 40/40.
Now the average gain for Houdini going from 32 bit to 64 bit on the four lists (comparing like version to like version) is 34.6 elo.
Conclusion: We must gain about 35 elo over Komodo 3 to pass Houdini at 40/40. We aren't quite there yet, but might be very soon.
You give new meaning to the phrase "There's lies, damned lies, and statistics".
The 95% confidence interval on each of your listed values above is easily 30 points...
Your numbers suggest that Houdini 1.5 32-bit is about 20 Elo stronger than Komodo 3 64-bit, and that most likely the 40/40 CCRL value is a statistical outlier.
Robert
Last edited by Houdini on Sun Dec 04, 2011 7:45 pm, edited 1 time in total.
Houdini wrote:
Strelka 5 is Houdini 1.5 RE, not Houdini 2.
I share your fears, any #1 engine will be RE'd in a matter of months. For Houdini 1.5 it's taken about 6 months .
Thx for the carification, so 6 month was the maximum, maybe he started just 4 weeks before release ... who knows?
Werewolf wrote:Larry,
What is your & Don's release cycle plans for Komodo?
Do you plan to follow the once-per-year model or the Rybka 2 model of a release every few months?
Personally, I think for serious chess players the Rybka 2 model was the best of all.
I think we will release far more often than once a year, maybe something like the Rybka 2 model. My feeling is that we should make a release whenever we have a significant improvement, at least enough that no one will be able to claim that it was not an improvement.
lkaufman wrote:Actually, I realized there is no great need to run this, because we can get the relative ratings at different time controls from the rating lists. Here is the data. Whenever both Houdini 2.0 and 1.5 appear on the same rating list in 32 bit, I averaged the ratings to get a bigger sample, since overall results for the two versions show hardly any difference (1 elo averaging the four lists cited). I also average Komodo 64 bit SSE and non-SSE when both appear on the same lists for the same reasons.
Here are the differences, Komodo 3 64 bit minus Houdini 64 bit:
40/4 (CEGT) : -35 (no CCRL data for H32 at this speed)
40/10(SWCR): -21
40/20 (CEGT): -15
40/40 (CCRL): +3
Notice the trend? It seems the point at which Komodo 3 pulls ahead appears to be at or just below 40/40.
Now the average gain for Houdini going from 32 bit to 64 bit on the four lists (comparing like version to like version) is 34.6 elo.
Conclusion: We must gain about 35 elo over Komodo 3 to pass Houdini at 40/40. We aren't quite there yet, but might be very soon.
You give new meaning to the phrase "There's lies, damned lies, and statistics".
The 95% confidence interval on each of your listed values above is easily 30 points...
Your numbers suggest that Houdini 1.5 32-bit is about 20 Elo stronger than Komodo 3 64-bit, and that most likely the 40/40 CCRL value is a statistical outlier.
Robert
OK, to be fair I compared Houdini 64 bit directly with Komodo 3 64 bit using the identical procedure as above, so a whole new set of data for Houdini. Here are the results:
40/4 (CEGT) : -71 (average of -68 for CCRL and -74 for CEGT)
40/10(SWCR): -48
40/20 (CEGT): -50.5
40/40 (CCRL): -37.5
Again, the same trend, almost the same decline in gap from top to bottom (38 elo for 1st table, 33.5 elo for second), and again the conclusion that we need about 35 elo to catch you at 40/40. Coincidence?
Most unlikely.
lkaufman wrote:Actually, I realized there is no great need to run this, because we can get the relative ratings at different time controls from the rating lists. Here is the data. Whenever both Houdini 2.0 and 1.5 appear on the same rating list in 32 bit, I averaged the ratings to get a bigger sample, since overall results for the two versions show hardly any difference (1 elo averaging the four lists cited). I also average Komodo 64 bit SSE and non-SSE when both appear on the same lists for the same reasons.
Here are the differences, Komodo 3 64 bit minus Houdini 64 bit:
40/4 (CEGT) : -35 (no CCRL data for H32 at this speed)
40/10(SWCR): -21
40/20 (CEGT): -15
40/40 (CCRL): +3
Notice the trend? It seems the point at which Komodo 3 pulls ahead appears to be at or just below 40/40.
Now the average gain for Houdini going from 32 bit to 64 bit on the four lists (comparing like version to like version) is 34.6 elo.
Conclusion: We must gain about 35 elo over Komodo 3 to pass Houdini at 40/40. We aren't quite there yet, but might be very soon.
You give new meaning to the phrase "There's lies, damned lies, and statistics".
The 95% confidence interval on each of your listed values above is easily 30 points...
Your numbers suggest that Houdini 1.5 32-bit is about 20 Elo stronger than Komodo 3 64-bit, and that most likely the 40/40 CCRL value is a statistical outlier.
Robert
OK, to be fair I compared Houdini 64 bit directly with Komodo 3 64 bit using the identical procedure as above, so a whole new set of data for Houdini. Here are the results:
40/4 (CEGT) : -71 (average of -68 for CCRL and -74 for CEGT)
40/10(SWCR): -48
40/20 (CEGT): -50.5
40/40 (CCRL): -37.5
Again, the same trend, almost the same decline in gap from top to bottom (38 elo for 1st table, 33.5 elo for second), and again the conclusion that we need about 35 elo to catch you at 40/40. Coincidence?
Most unlikely.
Hey Larry,
I know you liked the analogy to Moby Dick BUT I didn't really mean it as a compliment and am I wrong or doesn't the whale win?
Houdini wrote:
Strelka 5 is Houdini 1.5 RE, not Houdini 2.
I share your fears, any #1 engine will be RE'd in a matter of months. For Houdini 1.5 it's taken about 6 months.
Robert
You mean 6 months until Strelka has been released to public. You do not know how much time did Yuri Osipov spent on RE, neither when he did start his RE project. For me it took barely one week. As Milos S. pointed out, you made RE very very easy because of starting your project from Robbolito codebase... Personally I see no problem with that (other than you not telling the truth).
mwyoung wrote:[
Don and Larry needs to meet one of their deadlines and release Komodo 4 for testing.
All they are waiting for is a version of Komodo 4 that can overtake Houdini in the rating lists. With all the trash talk from them about Houdini's problems. Why the delay?
Basically, our problem is that we have two simultaneous projects going on. One is to come out with an MP version of Komodo as of the date code was frozen for it, and the other is the continued improvement of the SP version, which will eventually migrate to the MP.
I can propose an experiment you or anyone else out there with some good hardware can do. Run a long match of Komodo 3 64-bit SSE against Houdini (1.5 or 2.0, take your pick) 32-bit at some fairly long time control, maybe something like 25 minutes plus 15 seconds increment (i.e. five times what IPON uses), preferably with Ponder off so you can run twice as many at once. The point is that our latest Komodo version is roughly improved over Komodo 3 by the same amount that Houdini 64-bit outrates Houdini 32-bit. So whatever the result of such a match might be (I haven't run it so I don't know), it should be a good predictor for how our latest version will do against Houdini 64 bit at whatever time control you use. I know that Houdini 32-bit kills Komodo 3 64 bit at bullet chess (1' per game or anything comparably fast), so my expectation of a close result at long time controls rests squarely on my belief that Komodo scales much better than Houdini. I hope someone runs this test and posts the results here. But please, we need a few hundred games, not 50 or so. Of course if several people each run 50, that works fine!
Running a match now. With 5/6 TC. Perfect 12 abk. both engines.
Houdini 1.5 w32. vs. Komodo 3 -32. result so far. 9-0-9.
Houdini is killing Komodo after 18 games. Komodo has not won one game yet.
Best,
Gerold.
I understood Komodo 3 64 bit SSE - Houdini 1.5a 32 bit.
Of course and also long time control.
You are free not to do it but if you respond to Larry who asked to do testing of Komodo3 64 bit SSE against Houdini 1.5a 32 bit at long time control then I do not see the point of giving blitz results with komodo 32 bits.
I was going to run 100 plus games but since no one wants them run at blitz i stopped at 30 games. Komodo 3 won one game out of 30 played. Looks like Komodo needs a little more work.
Best,
Gerold.
You missed the whole point. We know that Komodo 3 32 bit is relatively very weak, just look at the rating lists. It is the product of a few minutes' work, we gave it zero priority (that may change in the future). The point was to use Komodo 64 bit as running Houdini 32 bit handicaps it by about the amount we have improved Komodo by since Komodo 3. That will tell you roughly where we stand right now. We'll still lose at blitz I think, though narrowly, but at something like the suggested 25' + 15" we might well come out ahead in a long match. Anyway I'd like to know.
Ok. First time i fine someone don't want their engine tested with equal conditions for both engines to find out how to improve their engine. I have deleted the games and Komodo from my machine. No problem.
Best,
Gerold.
Your conclusion is wrong.
The point is that CEGT and CCRL already tested in equal conditions so the fact that houdini is stronger than komodo in equal conditions is known and games are known.
Larry explained the interest in unequal conditions and his point is that Komodo3 64 bits is at similiar level to Komodo beta 32 bits,
so testing komodo3 64 bits against houdini 32 bits is going to give similiar results to testing komodo beta 32 bits against Houdini1.5 32 bits that is equal conditions.
You do not have to test komodo but it is better not to respond without reading.
