15,0000 game Komodo 11.01 test results

[PaulieD]

15000 GAME GAUNTLET

i7 6700HQ | 4 @ 2.6-3.5 | Win10x64-16GB |Gui~CuteChess 0.9.4
Book~TopGM_2move.pgn | Hash~8MB | Cores~1 | HT~On
Concurrency 7 | Ponder~No | TB~No | Time~20"+.2" (~66 secs)

Latest ASM +132 Elo at this time control

Code: Select all

   # PLAYER            :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 SF 170522ASMPD    :    3266      5    3000  67.8  1396  1278   326  42.6
   2 Komodo 11.01      :    3134      5    3000  32.2   326  1278  1396  42.6

Latest SF Dev +94 Elo at this time control

Code: Select all

   # PLAYER          :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 SF 170523       :    3247      5    3000  63.0  1213  1354   433  45.1
   2 Komodo 11.01    :    3153      5    3000  37.0   433  1354  1213  45.1

Seems SF8 is +80 Elo in 1 minute games.

Code: Select all

   # PLAYER          :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 Stockfish 8     :    3240      4    3000  61.2  1100  1473   427  49.1
   2 Komodo 11.01    :    3160      4    3000  38.8   427  1473  1100  49.1

H5.01 +56 Elo at this time control

Code: Select all

   # PLAYER              :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 Houdini 5.01 Pro    :    3228      4    3000  57.7  1006  1449   545  48.3
   2 Komodo 11.01        :    3172      4    3000  42.3   545  1449  1006  48.3

+10 Elo over K10.4

Code: Select all

   # PLAYER          :  RATING  ERROR  PLAYED   (%)    W     D    L  D(%)
   1 Komodo 11.01    :    3205      4    3000  51.3  679  1722  599  57.4
   2 Komodo 10.4     :    3195      4    3000  48.7  599  1722  679  57.4

[yanquis1972]

the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?

[mjlef]

the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?

We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.

Mark

[PaulieD]

the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?

We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.

Mark

This time control is not 20 seconds. It is the mid level time control used on the Stockfish Testing Framework. 10 + .1 is STC, 20 + .2 is MTC, and 60 + .6 is LTC.
All engine settings were at default.
Using the additional time increment .2 lengthens the games to about 66 seconds each.
That is not a "super fast" time control on todays hardware.

[David Xu]

the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?

We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.

Mark

This time control is not 20 seconds. It is the mid level time control used on the Stockfish Testing Framework. 10 + .1 is STC, 20 + .2 is MTC, and 60 + .6 is LTC.
All engine settings were at default.
Using the additional time increment .2 lengthens the games to about 66 seconds each.
That is not a "super fast" time control on todays hardware.

It depends on what you're comparing the time control to. Perhaps in SF testing, 20 + .2 testing counts as MTC, but most rating lists don't test at such short TC's. (Nor, for that matter, are most engine games played at such short TC's outside of testing.)

Also, the default settings of Komodo include Contempt being set at 10, which is known to hurt against roughly equal opponents such as Houdini and Stockfish. So yeah, that might contribute toward explaining this result. (Mark, Larry, if you're reading this, I would definitely recommend setting the default Contempt to 0 in future releases of Komodo, just to avoid stuff like this happening.)

[lkaufman]

I have run many thousands of fast games using LittleBlitzer in Windows at time limits like 1' + 1", 2' + 1", and 5' + 1", between Stockfish 8 and Komodo 11.01, and between Houdini 5.01 and Komodo 11.01, all single thread tests with hyperthreading on on 20 and 24 core machines, concurrency 32 and 48 respectively. These are using defaults, i.e. Contempt = 10 (probably with Contempt zero they would be a few elo better). The results have been remarkably consistent. We lose to SF8 by about 32 elo, and we barely beat Houdini (average plus 1 elo). I think that the amount of increment is the main factor in our results against these engines; the larger it is, the better we do. Base time doesn't seem to matter much. This would explain why the results in this thread are so poor with 0.2" inc. or even zero inc. in the case of 1 min results. Seems like one second increment is about the minimum to have meaningful results. I think this may also be why Komodo has done relatively poorly in games at 40 min plus 4 sec; it's still blitz with a 4 sec increment. I'm not sure why the increment seems so important for Komodo to shine. I'd like to see some tests with the FIDE standard 30" inc. (i.e. 90 min plus 30 sec). I note that Komodo always does well in CCRL and CEGT testing, which is more like testing with a large increment than a small one.

[Gusev]

Too many time forfeits at such short TCs?

the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?

[PaulieD]

You will get time forfeits with Little Blitzer, but not one with Cute Chess Gui 0.9.4.
There was nothing wrong with the settings/conditions. All games were with 20 +.2 for 66 secs. I used the term one minute in one set of games looslely.
They are duplicateable with the same conditions/settings.

Komodo may need a lot more time and cores than this time control...a lot more.

[shrapnel]

(Mark, Larry, if you're reading this, I would definitely recommend setting the default Contempt to 0 in future releases of Komodo, just to avoid stuff like this happening.)

Mark and Larry may or may not have got the Contempt Setting right, but they CERTAINLY got the Time Management settings spot on !
Yesterday I played 2 consecutive 5'+0 games on Infinity Chess with 2 different opponents, while using Komodo 11.01.
In both games, end-game Position was about equal, with 0.00 score.
Only difference was that Komodo had used up much less time than my opponents who were using Stockfish/Brainfish.
In fact, Komodo seemed to be aware in both games that Stockfish/Brainfish was in time trouble !

Komodo simply played faster and faster, without falling into the Trap of "Threefold-Repetition".
In one game, I remember, it simply moved a Bishop back and forth very fast, each time to a different square !
Needless to say, Stockfish/Brainfish Timed Out in both games!
So, maybe not a fair win, but as they say in chess and war, a win is a win, no matter HOW it is achieved.

[shrapnel]

A game Komodo won against the over-rated asmfish.....
[Event "Blitz 4' + 2'', Rated"]
[Site ""]
[Date "2017.5.26"]
[Round ""]
[White "shrapnel"]
[Black "ocirema"]
[Result "1-0"]
[ECO "C53"]
[WhiteElo "2472"]
[BlackElo "2612"]
[WhiteEngine "komodo-11.01-64bit_CTG"]
[BlackEngine "asmFishW_2017-05-22_popcnt"]
[EventType ""]
[WhiteTeam ""]
[BlackTeam ""]
[PlyCount "149"]
[TimeControl "240+2"]

[pgn]1.e4 {[%emt 0]} e5 {[%emt 1][%B]} 2.Nf3 {[%emt 0]} Nc6 {[%emt 0][%B]} 3.Bc4 {[%emt 0]} Bc5 {[%emt 0][%B]} 4.c3 {[%emt 0]} Nf6 {[%emt 0][%B]} 5.d3 {[%emt 0]} d6 {[%emt 0][%B]} 6.O-O {[%emt 0]} a5 {[%emt 1][%B]} 7.Bg5 {[%emt 0]} O-O {[%emt 1][%B]} 8.a4 {[%emt 0]} h6 {[%emt 1][%B]} 9.Bh4 {[%emt 0]} g5 {[%emt 1][%B]} 10.Bg3 {[%emt 0]} Kg7 {[%emt 1][%B]} 11.Nbd2 {[%emt 0]} g4 {[%emt 0]} 12.Nh4 {[%emt 0]} Nh5 { [%eval 0,02/17][%emt 0]} 13.Bb5 {[%emt 0]} Qg5 { [%eval 0,13/18][%emt 0]} 14.Nc4 {[%emt 0]} Nxg3 { [%eval 0,27/19][%emt 0]} 15.hxg3 {[%emt 0]} Ne7 { [%eval 0,27/19][%emt 0]} 16.d4 {[%emt 0]} Ba7 { [%eval 0,25/18][%emt 0]} 17.Ne3 {[%emt 0]} f5 { [%eval 0,38/18][%emt 1]} 18.exf5 {[%emt 0]} Nxf5 { [%eval 0,44/16][%emt 0]} 19.Nexf5+ {[%emt 0]} Bxf5 { [%eval -0,03/20][%emt 0]} 20.Be2 {[%emt 0]} h5 { [%eval 0,25/18][%emt 0]} 21.dxe5 {[%emt 0]} Bh7 { [%eval 0,10/19][%emt 0]} 22.exd6 {[%emt 0]} cxd6 { [%eval 0,15/20][%emt 0]} 23.Kh2 {[%emt 0]} Rxf2 { [%eval 0,27/20][%emt 0]} 24.Rxf2 {[%emt 0]} Bxf2 { [%eval 0,13/21][%emt 0]} 25.Qxd6 {[%emt 0]} Re8 { [%eval 0,09/24][%emt 0]} 26.Bc4 {[%emt 0]} Qe5 { [%eval 0,08/25][%emt 0]} 27.Qxe5+ {[%emt 0]} Rxe5 { [%eval 0.00/23][%emt 0]} 28.Rf1 {[%emt 0]} Be1 { [%eval 0,08/23][%emt 0]} 29.Rf7+ {[%emt 0]} Kh6 { [%eval 0,08/24][%emt 0]} 30.Rxb7 {[%emt 1]} Re3 { [%eval 0,08/27][%emt 0]} 31.Rb6+ {[%emt 4]} Kg5 { [%eval 1/45][%emt 2:02]} 32.Rb5+ {[%emt 6]} Kf6 { [%eval 1/45][%emt 11]} 33.Rb6+ {[%emt 0]} Ke7 { [%eval 1/45][%emt 1:11]} 34.Rb7+ {[%emt 6]} Kf6 { [%eval 1/43][%emt 15]} 35.Nf3 {[%emt 0]} gxf3 { [%eval 0,85/31][%emt 12]} 36.Rxh7 {[%emt 0]} f2 { [%eval 0,91/33][%emt 6]} 37.Rf7+ {[%emt 18]} Kg5 { [%eval 0,98/33][%emt 4]} 38.Ra7 {[%emt 6]} Re5 { [%eval 1/32][%emt 0]} 39.Ra8 {[%emt 10]} Kh6 { [%eval 0,92/36][%emt 25]} 40.Rd8 {[%emt 26]} Re4 { [%eval 0,37/30][%emt 9]} 41.Rd6+ {[%emt 5]} Kg7 { [%eval 0,57/31][%emt 0]} 42.Rd4 {[%emt 3]} Rg4 { [%eval 0,77/41][%emt 16]} 43.Bf1 {[%emt 0]} Rxd4 { [%eval 0,84/26][%emt 1]} 44.cxd4 {[%emt 4]} Kg6 { [%eval 0,84/46][%emt 7]} 45.d5 {[%emt 4]} Bb4 { [%eval 1/43][%emt 8]} 46.g4 {[%emt 0]} h4 { [%eval 1/42][%emt 4]} 47.Kh3 {[%emt 17]} Kg5 { [%eval 1/57][%emt 0]} 48.g3 {[%emt 5]} hxg3 { [%eval 1/59][%emt 0]} 49.Kxg3 {[%emt 3]} Bd6+ { [%eval 1/54][%emt 0]} 50.Kf3 {[%emt 3]} Be7 { [%eval 1/50][%emt 5]} 51.Be2 {[%emt 30]} Bc5 { [%eval 1/53][%emt 0]} 52.Ke4 {[%emt 16]} Kf6 { [%eval 1,58/32][%emt 9]} 53.b3 {[%emt 2]} Bd6 { [%eval 2,06/28][%emt 4]} 54.Kd4 {[%emt 29]} Ke7 { [%eval 5,58/30][%emt 1]} 55.g5 {[%emt 8]} Bf4 { [%eval 7,22/28][%emt 0]} 56.g6 {[%emt 24]} Kf6 { [%eval 14,56/27][%emt 0]} 57.Bd3 {[%emt 4]} f1=R { [%eval 17,64/24][%emt 2]} 58.Bxf1 {[%emt 4]} Kxg6 { [%eval 21,32/25][%emt 2]} 59.b4 {[%emt 0]} Bd2 { [%eval 59,29/22][%emt 6]} 60.bxa5 {[%emt 5]} Bxa5 { [%eval 128,41/32][%emt 3]} 61.Kc4 {[%emt 3]} Kf6 { [%eval 128,43/34][%emt 3]} 62.Kb5 {[%emt 1]} Bc7 { [%eval 128,44/43][%emt 1]} 63.a5 {[%emt 1]} Bh2 { [%eval 128,45/44][%emt 0]} 64.a6 {[%emt 1]} Bg1 { [%eval 128,46/45][%emt 0]} 65.d6 {[%emt 1]} Bf2 { [%eval 128,47/47][%emt 3]} 66.Kc6 {[%emt 0]} Kf7 { [%eval 128,47/48][%emt 0]} 67.d7 {[%emt 0]} Ke7 { [%eval 128,48/54][%emt 2]} 68.Kc7 {[%emt 0]} Kf7 { [%eval 128,49/44][%emt 2]} 69.d8=Q {[%emt 1]} Bb6+ { [%eval 128,50/70][%emt 3]} 70.Kxb6 {[%emt 1]} Kg6 { [%eval #5/127][%emt 2]} 71.a7 {[%emt 0]} Kf5 { [%eval #4/92][%emt 1]} 72.a8=Q {[%emt 0]} Kg6 { [%eval #3/127][%emt 1]} 73.Bd3+ {[%emt 0]} Kf7 { [%eval #2/127][%emt 1]} 74.Qf3+ {[%emt 0]} Ke6 { [%eval #1/127][%emt 1]} 75.Qfd5# {Mated[%emt 0]} 1-0[/pgn]