Komodo 9.2 x64 12CPU at CEGT 40/4

[Leto]

Running a 750 game gauntlet right now, and then I'll run 200 more games with Stockfish 6 12CPU and 50 more games to Houdini 4 4CPU to match the games played by K9.1 12CPU.

Here's some interim results after 206 games, I've included K9.1 12CPU's scores in () after 9.2's score:

Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 12 7.0 - 7.0 +3/=8/-3 50.00% (K9.1 55%)
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 8 7.0 - 7.0 +2/=10/-2 50.00% (K9.1 56%)
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 12 8.5 - 5.5 +5/=7/-2 60.71% (K9.1 64%)
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 4 10.5 - 3.5 +7/=7/-0 75.00% (K9.1 60%)
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 8 8.5 - 5.5 +4/=9/-1 60.71% (K9.1 67%)
Komodo 9.2 64-bit - Houdini 3 Pro x64 12 10.0 - 4.0 +6/=8/-0 71.43% (K9.1 66%)
Komodo 9.2 64-bit - Houdini 4 Pro x64 12a 10.0 - 4.0 +9/=2/-3 71.43% (K9.1 78%)
Komodo 9.2 64-bit - Houdini 4 Pro x64 8 9.5 - 4.5 +6/=7/-1 67.86% (K9.1 80%)
Komodo 9.2 64-bit - Gull 3 x64 12 10.5 - 3.5 +8/=5/-1 75.00% (K9.1 77%)
Komodo 9.2 64-bit - Houdini 4 Pro x64 4a 11.0 - 2.0 +9/=5/-0 82.14% (K9.1 76%)
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 1 9.5 - 3.5 +6/=7/-0 73.08% (K9.1 70%)
Komodo 9.2 64-bit - Gull 3 x64 4 10.0 - 3.0 +7/=6/-0 76.92% (K9.1 74%)
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 1 10.0 - 3.0 +7/=6/-0 76.92% (K9.1 74%)
Komodo 9.2 64-bit - Deep Rybka 4.1 SSE42 x64 12 10.0 - 3.0 +7/=6/-0 76.92% (K9.1 87%)
Komodo 9.2 64-bit - Critter 1.6 64-bit 4 11.0 - 2.0 +9/=4/-0 84.62% (K9.1 84%)

Global score after 206 games = 69.9%, K9.1 x64 12CPU has 68.2% after 1000 games.

I might make a 0 contempt run after I do 1000 games for the default version, I'll see if I'm allowed to include it on the list, if not I'll just post the results here.

CEGT 40/4 rating list:
http://www.husvankempen.de/nunn/40_4_Ra ... liste.html

[Leto]

Results after 358 games out of 750:

k92 gauntlet 2015

Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 12 13.0 - 11.0 +7/=12/-5 54.17%
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 8 12.0 - 12.0 +5/=14/-5 50.00%
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 12 15.5 - 8.5 +9/=13/-2 64.58%
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 4 17.5 - 6.5 +11/=13/-0 72.92%
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 8 14.0 - 10.0 +7/=14/-3 58.33%
Komodo 9.2 64-bit - Houdini 3 Pro x64 12 18.5 - 5.5 +13/=11/-0 77.08%
Komodo 9.2 64-bit - Houdini 4 Pro x64 12a 17.5 - 6.5 +15/=5/-4 72.92%
Komodo 9.2 64-bit - Houdini 4 Pro x64 8 18.0 - 6.0 +13/=10/-1 75.00%
Komodo 9.2 64-bit - Gull 3 x64 12 17.0 - 7.0 +12/=10/-2 70.83%
Komodo 9.2 64-bit - Houdini 4 Pro x64 4a 19.0 - 5.0 +15/=8/-1 79.17%
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 1 16.5 - 7.5 +9/=15/-0 68.75%
Komodo 9.2 64-bit - Gull 3 x64 4 18.5 - 5.5 +14/=9/-1 77.08%
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 1 18.5 - 5.5 +13/=11/-0 77.08%
Komodo 9.2 64-bit - Deep Rybka 4.1 SSE42 x64 12 19.0 - 4.0 +15/=8/-0 82.61%
Komodo 9.2 64-bit - Critter 1.6 64-bit 4 20.5 - 2.5 +18/=5/-0 89.13%

Global score after 358 games = 71%, K9.1 x64 12CPU has 68.2% after 1000 games.

[Leto]

First gauntlet completed:

k92 gauntlet 2015

Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 12 26.0 - 24.0 +12/=28/-10 52.00%
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 8 26.5 - 23.5 +12/=29/-9 53.00%
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 12 31.5 - 18.5 +17/=29/-4 63.00%
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 4 32.0 - 18.0 +16/=32/-2 64.00%
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 8 32.5 - 17.5 +20/=25/-5 65.00%
Komodo 9.2 64-bit - Houdini 3 Pro x64 12 39.5 - 10.5 +30/=19/-1 79.00%
Komodo 9.2 64-bit - Houdini 4 Pro x64 12a 35.5 - 14.5 +27/=17/-6 71.00%
Komodo 9.2 64-bit - Houdini 4 Pro x64 8 39.5 - 10.5 +32/=15/-3 79.00%
Komodo 9.2 64-bit - Gull 3 x64 12 33.5 - 16.5 +21/=25/-4 67.00%
Komodo 9.2 64-bit - Houdini 4 Pro x64 4a 40.5 - 9.5 +32/=17/-1 81.00%
Komodo 9.2 64-bit - Stockfish 6 64 POPCNT 1 36.5 - 13.5 +24/=25/-1 73.00%
Komodo 9.2 64-bit - Gull 3 x64 4 38.0 - 12.0 +28/=20/-2 76.00%
Komodo 9.2 64-bit - Stockfish 5 64 SSE4.2 1 39.0 - 11.0 +29/=20/-1 78.00%
Komodo 9.2 64-bit - Deep Rybka 4.1 SSE42 x64 12 43.0 - 7.0 +36/=14/-0 86.00%
Komodo 9.2 64-bit - Critter 1.6 64-bit 4 44.5 - 5.5 +39/=11/-0 89.00%

Global score after 750 games = 71%, K9.1 x64 12CPU has 68.2% after 1000 games.

[Leto]

After 1000 games played by Komodo 9.2 x64 12CPU:

k92 gauntlet 2015

Komodo 9.2 x64 12CPU - Stockfish 6.0 x64 12CPU 132.5 - 117.5 +54/=157/-39 53.00%
Komodo 9.2 x64 12CPU - Stockfish 6.0 x64 8CPU 26.5 - 23.5 +12/=29/-9 53.00%
Komodo 9.2 x64 12CPU - Stockfish 5.0 x64 12CPU 31.5 - 18.5 +17/=29/-4 63.00%
Komodo 9.2 x64 12CPU - Stockfish 6.0 x64 4CPU 32.0 - 18.0 +16/=32/-2 64.00%
Komodo 9.2 x64 12CPU - Stockfish 5.0 x64 8CPU 32.5 - 17.5 +20/=25/-5 65.00%
Komodo 9.2 x64 12CPU - Houdini 3.0 x64 12CPU 39.5 - 10.5 +30/=19/-1 79.00%
Komodo 9.2 x64 12CPU - Houdini 4.0 x64 12CPU 35.5 - 14.5 +27/=17/-6 71.00%
Komodo 9.2 x64 12CPU - Houdini 4.0 x64 8CPU 39.5 - 10.5 +32/=15/-3 79.00%
Komodo 9.2 x64 12CPU - Gull 3.0 x64 12CPU 33.5 - 16.5 +21/=25/-4 67.00%
Komodo 9.2 x64 12CPU - Houdini 4.0 x64 4CPU 82.5 - 17.5 +67/=31/-2 82.50%
Komodo 9.2 x64 12CPU - Stockfish 6.0 x64 1CPU 36.5 - 13.5 +24/=25/-1 73.00%
Komodo 9.2 x64 12CPU - Gull 3.0 x64 4CPU 38.0 - 12.0 +28/=20/-2 76.00%
Komodo 9.2 x64 12CPU - Stockfish 5.0 x64 1CPU 39.0 - 11.0 +29/=20/-1 78.00%
Komodo 9.2 x64 12CPU - Rybka 4.1 x64 12CPU 43.0 - 7.0 +36/=14/-0 86.00%
Komodo 9.2 x64 12CPU - Critter 1.6 x64 4CPU 44.5 - 5.5 +39/=11/-0 89.00%

global score for Komodo 9.2 x64 12CPU default is 686.5/1000 (68.6%)

After 750 games from Komodo 9.2 x64 C0 12CPU contempt 0:

K92 0 contempt gauntlet 1 2015

Komodo 9.2 x64 C0 12CPU - Stockfish 6.0 x64 12CPU 30.5 - 19.5 +15/=31/-4 61.00%
Komodo 9.2 x64 C0 12CPU - Stockfish 6.0 x64 8CPU 27.5 - 22.5 +9/=37/-4 55.00%
Komodo 9.2 x64 C0 12CPU - Stockfish 5.0 x64 12CPU 31.0 - 19.0 +16/=30/-4 62.00%
Komodo 9.2 x64 C0 12CPU - Stockfish 6.0 x64 4CPU 31.0 - 19.0 +15/=32/-3 62.00%
Komodo 9.2 x64 C0 12CPU - Stockfish 5.0 x64 8CPU 36.0 - 14.0 +24/=24/-2 72.00%
Komodo 9.2 x64 C0 12CPU - Houdini 3.0 x64 12CPU 38.5 - 11.5 +29/=19/-2 77.00%
Komodo 9.2 x64 C0 12CPU - Houdini 4.0 x64 12CPU 37.0 - 13.0 +26/=22/-2 74.00%
Komodo 9.2 x64 C0 12CPU - Houdini 4.0 x64 8CPU 34.5 - 15.5 +23/=23/-4 69.00%
Komodo 9.2 x64 C0 12CPU - Gull 3.0 x64 12CPU 32.0 - 18.0 +16/=32/-2 64.00%
Komodo 9.2 x64 C0 12CPU - Houdini 4.0 x64 4CPU 43.0 - 7.0 +37/=12/-1 86.00%
Komodo 9.2 x64 C0 12CPU - Stockfish 6.0 x64 1CPU 37.0 - 13.0 +24/=26/-0 74.00%
Komodo 9.2 x64 C0 12CPU - Gull 3.0 x64 4CPU 37.5 - 12.5 +27/=21/-2 75.00%
Komodo 9.2 x64 C0 12CPU - Stockfish 5.0 x64 1CPU 41.0 - 9.0 +32/=18/-0 82.00%
Komodo 9.2 x64 C0 12CPU - Rybka 4.1 x64 12CPU 42.5 - 7.5 +35/=15/-0 85.00%
Komodo 9.2 x64 C0 12CPU - Critter 1.6 x64 4CPU 42.0 - 8.0 +35/=14/-1 84.00%

global score for Komodo 9.2 x64 C0 12CPU contempt 0 is 541/750 (72.1%)

I'm very surprised the 0 contempt version outperformed the default version (contempt 15). Maybe something not right with default contempt setting? I will of course run the extra 200 games against Stockfish 6 x64 12CPU and 50 games with Houdini 4 x64 4CPU to match the games played by the default version but it looks like the contempt 0 version will end up with the higher score.

[JJJ]

Contempt 15 is too much against Houdini Gull Stockfish.
Ok against Rybka Critter.

It is not the first test who does show it.

[lkaufman]

Contempt 15 is too much against Houdini Gull Stockfish.
Ok against Rybka Critter.

It is not the first test who does show it.

Contempt 15 is certainly too much against Stockfish (maybe 5 for SF6 and zero for latest SF), but is probably pretty close to correct against Houdini and Gull, though maybe ten would be better against them. Against engines 200 to 300 below Komdo probably 20 or even 25 would do best. 15 is some sort of compromise.

[Leto]

Contempt 15 is too much against Houdini Gull Stockfish.
Ok against Rybka Critter.

It is not the first test who does show it.

Contempt 15 is certainly too much against Stockfish (maybe 5 for SF6 and zero for latest SF), but is probably pretty close to correct against Houdini and Gull, though maybe ten would be better against them. Against engines 200 to 300 below Komdo probably 20 or even 25 would do best. 15 is some sort of compromise.

I don't know how the 0 contempt setting does on less cores (like 1 or 4 cores) but if it does as well as it does on 12 cores why not use 0 contempt as the default setting?

[lkaufman]

Contempt 15 is too much against Houdini Gull Stockfish.
Ok against Rybka Critter.

It is not the first test who does show it.

Contempt 15 is certainly too much against Stockfish (maybe 5 for SF6 and zero for latest SF), but is probably pretty close to correct against Houdini and Gull, though maybe ten would be better against them. Against engines 200 to 300 below Komdo probably 20 or even 25 would do best. 15 is some sort of compromise.

I don't know how the 0 contempt setting does on less cores (like 1 or 4 cores) but if it does as well as it does on 12 cores why not use 0 contempt as the default setting?

I'll have to see how the ratings of komodo 9.2 and 9.1 compare on all the various rating lists with various numbers of cores. Since I estimate the "pure" gain,excluding the effect of contempt, to be in the ten to fifteen elo range (depending on time control), If the actual average gain is more than that it would mean that contempt has helped.

[Leto]

200 additional games played between K9.2 x64 C0 12CPU and Stockfish 6 x64 12CPU, and 50 additional games played against Houdini 4 x64 4CPU.


1 Komodo 9.2 64-bit C0 +69 +53/=133/-14 59.75% 119.5/200
2 Stockfish 6 64 POPCNT 12 -69 +14/=133/-53 40.25% 80.5/200


1 Komodo 9.2 64-bit C0 +220 +30/=18/-2 78.00% 39.0/50
2 Houdini 4 Pro x64 4 -220 +2/=18/-30 22.00% 11.0/50

So after 250 games the 0 contempt version has scored 60.37% against Stockfish 6 12CPU (K9.2 default scored 53% with the same amount of games)

And after 100 games the 0 contempt version has scored 82% against Houdini 4 4CPU (K9.2 default scored 82.5% with the same amount of games).

And with 1000 games each from both the default and the 0 contempt version, the 0 contempt version ended up with 71.49% overall (default version ended up with 68.7%) .

The list will probably be updated within two weeks.

[lkaufman]

200 additional games played between K9.2 x64 C0 12CPU and Stockfish 6 x64 12CPU, and 50 additional games played against Houdini 4 x64 4CPU.


1 Komodo 9.2 64-bit C0 +69 +53/=133/-14 59.75% 119.5/200
2 Stockfish 6 64 POPCNT 12 -69 +14/=133/-53 40.25% 80.5/200


1 Komodo 9.2 64-bit C0 +220 +30/=18/-2 78.00% 39.0/50
2 Houdini 4 Pro x64 4 -220 +2/=18/-30 22.00% 11.0/50

So after 250 games the 0 contempt version has scored 60.37% against Stockfish 6 12CPU (K9.2 default scored 53% with the same amount of games)

And after 100 games the 0 contempt version has scored 82% against Houdini 4 4CPU (K9.2 default scored 82.5% with the same amount of games).

And with 1000 games each from both the default and the 0 contempt version, the 0 contempt version ended up with 71.49% overall (default version ended up with 68.7%) .

The list will probably be updated within two weeks.

This makes sense. Default contempt will clearly hurt against SF, although I'm surprised at the maginitude of the difference. Contempt makes Komodo try to avoid endgames, and it may be that much of our advantage over Stockfish is in the endgame. Against Houdini your result shows a trivial benefit for contempt, which also sounds right; probably a vaue of ten would be best against Houdini. Probably against all engines weaker than Houdini and Gull default contempt is a benefit. Presumably more games are played by testers against all such weaker engines combined than just against Stockfish versions, so on balance it should help. Perhaps we should have set the default to ten since so many people test against Stockfish. I may consider reducing the default to ten for the next release, unless we have enough other improvements to make this unnecessary.