CCRL 40/4 testing Houdini, Stockfish and Komodo

[Dr.Wael Deeb]

Dr.:

I give him maximum 2-3 months and he'll fall behind....

I assume you're referring to blitz and TC <60 wherein H4 is clearly the King. However, the tests at this and other sites at >60 TC show that the latest versions of SF have caught up if not surpassed H4.

So breaks of one year to release a new version won't be advised nowadays....Dr.D

Pretty sure Robert has long ago concluded same.

Exactly the case Master Bobby
Cheers,
Dr.D

[Dr.Wael Deeb]

Thanks for the test!
Three comments.

1) It certainly seems that H4 gains more from the syzygy bases...
The error levels in your ratings are 25 to 35 Elo, which is very large compared to the observed rating differences.
This makes all conclusions very tentative - your guess that engine A gains more from Syzygy than engine B involves a combination of 4 ratings, with a combined error level of 50 to 70 Elo.

2) "I assume that knowledge that is no longer required for the endgame has been removed speeding up the program. "
Based on a test with high error margins you entertain some wild speculation, which of course is incorrect.

3) If you plan on playing only with Houdini/Stockfish/Komodo you'd better use Contempt 0 for Houdini, it provides the highest level of play against more or less equal opponents.

Cheers,
Robert

Regarding your last point, the same would apply for Komodo; if you test Houdini with Contempt 0 against these opponents, you should do the same for the same reason with Komodo. Then you have to maintain separate ratings for contempt 0 for each engine, and then what happens when another tester uses a wider range of opponents? It's a real can of worms. Maybe all engines should always be tested with zero contempt.

Or maybe they should be tested with their default settings first and then adding additional personalities to the pole for more accurate results....

The other option is the really big can of worms that will be open as you stated Larry....
Dr.D

[Shaun]

3) If you plan on playing only with Houdini/Stockfish/Komodo you'd better use Contempt 0 for Houdini, it provides the highest level of play against more or less equal opponents.

Cheers,
Robert

Regarding your last point, the same would apply for Komodo; if you test Houdini with Contempt 0 against these opponents, you should do the same for the same reason with Komodo. Then you have to maintain separate ratings for contempt 0 for each engine, and then what happens when another tester uses a wider range of opponents? It's a real can of worms. Maybe all engines should always be tested with zero contempt.

For the rating list it makes sense to use the default, it's then up to the author to set the right default for varied opponents. For head to head matches I am happy to experiment with settings but typically* these will not be included in the rating lists.

Shaun

* EDIT: for clarity the results would not be combined however if there was enough interest I would run sufficient games to make a separate listing viable.

[Shaun]

In fact I agree with Mr Houdart that error levels are large in your tests, but not for the reasons he thinks !

Hi please expand... either here or via pm if you have any observations or concerns I should be aware of!!!

[Stefan Schiffermueller]

5 men Nalimov and 6 men syzygy on SSD
Shaun

The syzygy are on a fast SSD. So the default value for 'EGTB Probe Depth' is not optimal. Maybe Robert Houdert can give you a recommendation.