15,0000 game Komodo 11.01 test results

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
PaulieD
Posts: 205
Joined: Tue Jun 25, 2013 6:19 pm

15,0000 game Komodo 11.01 test results

Post by PaulieD » Thu May 25, 2017 1:11 pm

15000 GAME GAUNTLET

i7 6700HQ | 4 @ 2.6-3.5 | Win10x64-16GB |Gui~CuteChess 0.9.4
Book~TopGM_2move.pgn | Hash~8MB | Cores~1 | HT~On
Concurrency 7 | Ponder~No | TB~No | Time~20"+.2" (~66 secs)


Latest ASM +132 Elo at this time control

Code: Select all

   # PLAYER            :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 SF 170522ASMPD    :    3266      5    3000  67.8  1396  1278   326  42.6
   2 Komodo 11.01      :    3134      5    3000  32.2   326  1278  1396  42.6
Latest SF Dev +94 Elo at this time control

Code: Select all

   # PLAYER          :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 SF 170523       :    3247      5    3000  63.0  1213  1354   433  45.1
   2 Komodo 11.01    :    3153      5    3000  37.0   433  1354  1213  45.1
Seems SF8 is +80 Elo in 1 minute games.

Code: Select all

   # PLAYER          :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 Stockfish 8     :    3240      4    3000  61.2  1100  1473   427  49.1
   2 Komodo 11.01    :    3160      4    3000  38.8   427  1473  1100  49.1
H5.01 +56 Elo at this time control

Code: Select all

   # PLAYER              :  RATING  ERROR  PLAYED   (%)     W     D     L  D(%)
   1 Houdini 5.01 Pro    :    3228      4    3000  57.7  1006  1449   545  48.3
   2 Komodo 11.01        :    3172      4    3000  42.3   545  1449  1006  48.3
+10 Elo over K10.4

Code: Select all

   # PLAYER          :  RATING  ERROR  PLAYED   (%)    W     D    L  D(%)
   1 Komodo 11.01    :    3205      4    3000  51.3  679  1722  599  57.4
   2 Komodo 10.4     :    3195      4    3000  48.7  599  1722  679  57.4

yanquis1972
Posts: 1766
Joined: Tue Jun 02, 2009 10:14 pm

Re: 15,0000 game Komodo 11.01 test results

Post by yanquis1972 » Thu May 25, 2017 6:43 pm

the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?

mjlef
Posts: 1467
Joined: Thu Mar 30, 2006 12:08 pm
Contact:

Re: 15,0000 game Komodo 11.01 test results

Post by mjlef » Thu May 25, 2017 7:36 pm

yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.

Mark

PaulieD
Posts: 205
Joined: Tue Jun 25, 2013 6:19 pm

Re: 15,0000 game Komodo 11.01 test results

Post by PaulieD » Thu May 25, 2017 7:45 pm

mjlef wrote:
yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.

Mark
This time control is not 20 seconds. It is the mid level time control used on the Stockfish Testing Framework. 10 + .1 is STC, 20 + .2 is MTC, and 60 + .6 is LTC.
All engine settings were at default.
Using the additional time increment .2 lengthens the games to about 66 seconds each.
That is not a "super fast" time control on todays hardware.

David Xu
Posts: 47
Joined: Mon Oct 31, 2016 8:45 pm

Re: 15,0000 game Komodo 11.01 test results

Post by David Xu » Thu May 25, 2017 9:56 pm

PaulieD wrote:
mjlef wrote:
yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.

Mark
This time control is not 20 seconds. It is the mid level time control used on the Stockfish Testing Framework. 10 + .1 is STC, 20 + .2 is MTC, and 60 + .6 is LTC.
All engine settings were at default.
Using the additional time increment .2 lengthens the games to about 66 seconds each.
That is not a "super fast" time control on todays hardware.
It depends on what you're comparing the time control to. Perhaps in SF testing, 20 + .2 testing counts as MTC, but most rating lists don't test at such short TC's. (Nor, for that matter, are most engine games played at such short TC's outside of testing.)

Also, the default settings of Komodo include Contempt being set at 10, which is known to hurt against roughly equal opponents such as Houdini and Stockfish. So yeah, that might contribute toward explaining this result. (Mark, Larry, if you're reading this, I would definitely recommend setting the default Contempt to 0 in future releases of Komodo, just to avoid stuff like this happening.)

lkaufman
Posts: 4477
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: 15,0000 game Komodo 11.01 test results

Post by lkaufman » Thu May 25, 2017 11:59 pm

I have run many thousands of fast games using LittleBlitzer in Windows at time limits like 1' + 1", 2' + 1", and 5' + 1", between Stockfish 8 and Komodo 11.01, and between Houdini 5.01 and Komodo 11.01, all single thread tests with hyperthreading on on 20 and 24 core machines, concurrency 32 and 48 respectively. These are using defaults, i.e. Contempt = 10 (probably with Contempt zero they would be a few elo better). The results have been remarkably consistent. We lose to SF8 by about 32 elo, and we barely beat Houdini (average plus 1 elo). I think that the amount of increment is the main factor in our results against these engines; the larger it is, the better we do. Base time doesn't seem to matter much. This would explain why the results in this thread are so poor with 0.2" inc. or even zero inc. in the case of 1 min results. Seems like one second increment is about the minimum to have meaningful results. I think this may also be why Komodo has done relatively poorly in games at 40 min plus 4 sec; it's still blitz with a 4 sec increment. I'm not sure why the increment seems so important for Komodo to shine. I'd like to see some tests with the FIDE standard 30" inc. (i.e. 90 min plus 30 sec). I note that Komodo always does well in CCRL and CEGT testing, which is more like testing with a large increment than a small one.
Komodo rules!

Gusev
Posts: 1475
Joined: Mon Jan 28, 2013 1:51 pm

Re: 15,0000 game Komodo 11.01 test results

Post by Gusev » Fri May 26, 2017 12:18 am

Too many time forfeits at such short TCs?
yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?

PaulieD
Posts: 205
Joined: Tue Jun 25, 2013 6:19 pm

Re: 15,0000 game Komodo 11.01 test results

Post by PaulieD » Fri May 26, 2017 2:39 am

You will get time forfeits with Little Blitzer, but not one with Cute Chess Gui 0.9.4.
There was nothing wrong with the settings/conditions. All games were with 20 +.2 for 66 secs. I used the term one minute in one set of games looslely.
They are duplicateable with the same conditions/settings.

Komodo may need a lot more time and cores than this time control...a lot more.

shrapnel
Posts: 1340
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: 15,0000 game Komodo 11.01 test results

Post by shrapnel » Fri May 26, 2017 3:40 am

David Xu wrote: (Mark, Larry, if you're reading this, I would definitely recommend setting the default Contempt to 0 in future releases of Komodo, just to avoid stuff like this happening.)
Mark and Larry may or may not have got the Contempt Setting right, but they CERTAINLY got the Time Management settings spot on !
Yesterday I played 2 consecutive 5'+0 games on Infinity Chess with 2 different opponents, while using Komodo 11.01.
In both games, end-game Position was about equal, with 0.00 score.
Only difference was that Komodo had used up much less time than my opponents who were using Stockfish/Brainfish.
In fact, Komodo seemed to be aware in both games that Stockfish/Brainfish was in time trouble !
:)
Komodo simply played faster and faster, without falling into the Trap of "Threefold-Repetition".
In one game, I remember, it simply moved a Bishop back and forth very fast, each time to a different square !
Needless to say, Stockfish/Brainfish Timed Out in both games! :D
So, maybe not a fair win, but as they say in chess and war, a win is a win, no matter HOW it is achieved. :)
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

shrapnel
Posts: 1340
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: 15,0000 game Komodo 11.01 test results

Post by shrapnel » Fri May 26, 2017 11:36 am

A game Komodo won against the over-rated asmfish.....
[Event "Blitz 4' + 2'', Rated"]
[Site ""]
[Date "2017.5.26"]
[Round ""]
[White "shrapnel"]
[Black "ocirema"]
[Result "1-0"]
[ECO "C53"]
[WhiteElo "2472"]
[BlackElo "2612"]
[WhiteEngine "komodo-11.01-64bit_CTG"]
[BlackEngine "asmFishW_2017-05-22_popcnt"]
[EventType ""]
[WhiteTeam ""]
[BlackTeam ""]
[PlyCount "149"]
[TimeControl "240+2"]

[pgn]1.e4 {[%emt 0]} e5 {[%emt 1][%B]} 2.Nf3 {[%emt 0]} Nc6 {[%emt 0][%B]} 3.Bc4 {[%emt 0]} Bc5 {[%emt 0][%B]} 4.c3 {[%emt 0]} Nf6 {[%emt 0][%B]} 5.d3 {[%emt 0]} d6 {[%emt 0][%B]} 6.O-O {[%emt 0]} a5 {[%emt 1][%B]} 7.Bg5 {[%emt 0]} O-O {[%emt 1][%B]} 8.a4 {[%emt 0]} h6 {[%emt 1][%B]} 9.Bh4 {[%emt 0]} g5 {[%emt 1][%B]} 10.Bg3 {[%emt 0]} Kg7 {[%emt 1][%B]} 11.Nbd2 {[%emt 0]} g4 {[%emt 0]} 12.Nh4 {[%emt 0]} Nh5 { [%eval 0,02/17][%emt 0]} 13.Bb5 {[%emt 0]} Qg5 { [%eval 0,13/18][%emt 0]} 14.Nc4 {[%emt 0]} Nxg3 { [%eval 0,27/19][%emt 0]} 15.hxg3 {[%emt 0]} Ne7 { [%eval 0,27/19][%emt 0]} 16.d4 {[%emt 0]} Ba7 { [%eval 0,25/18][%emt 0]} 17.Ne3 {[%emt 0]} f5 { [%eval 0,38/18][%emt 1]} 18.exf5 {[%emt 0]} Nxf5 { [%eval 0,44/16][%emt 0]} 19.Nexf5+ {[%emt 0]} Bxf5 { [%eval -0,03/20][%emt 0]} 20.Be2 {[%emt 0]} h5 { [%eval 0,25/18][%emt 0]} 21.dxe5 {[%emt 0]} Bh7 { [%eval 0,10/19][%emt 0]} 22.exd6 {[%emt 0]} cxd6 { [%eval 0,15/20][%emt 0]} 23.Kh2 {[%emt 0]} Rxf2 { [%eval 0,27/20][%emt 0]} 24.Rxf2 {[%emt 0]} Bxf2 { [%eval 0,13/21][%emt 0]} 25.Qxd6 {[%emt 0]} Re8 { [%eval 0,09/24][%emt 0]} 26.Bc4 {[%emt 0]} Qe5 { [%eval 0,08/25][%emt 0]} 27.Qxe5+ {[%emt 0]} Rxe5 { [%eval 0.00/23][%emt 0]} 28.Rf1 {[%emt 0]} Be1 { [%eval 0,08/23][%emt 0]} 29.Rf7+ {[%emt 0]} Kh6 { [%eval 0,08/24][%emt 0]} 30.Rxb7 {[%emt 1]} Re3 { [%eval 0,08/27][%emt 0]} 31.Rb6+ {[%emt 4]} Kg5 { [%eval 1/45][%emt 2:02]} 32.Rb5+ {[%emt 6]} Kf6 { [%eval 1/45][%emt 11]} 33.Rb6+ {[%emt 0]} Ke7 { [%eval 1/45][%emt 1:11]} 34.Rb7+ {[%emt 6]} Kf6 { [%eval 1/43][%emt 15]} 35.Nf3 {[%emt 0]} gxf3 { [%eval 0,85/31][%emt 12]} 36.Rxh7 {[%emt 0]} f2 { [%eval 0,91/33][%emt 6]} 37.Rf7+ {[%emt 18]} Kg5 { [%eval 0,98/33][%emt 4]} 38.Ra7 {[%emt 6]} Re5 { [%eval 1/32][%emt 0]} 39.Ra8 {[%emt 10]} Kh6 { [%eval 0,92/36][%emt 25]} 40.Rd8 {[%emt 26]} Re4 { [%eval 0,37/30][%emt 9]} 41.Rd6+ {[%emt 5]} Kg7 { [%eval 0,57/31][%emt 0]} 42.Rd4 {[%emt 3]} Rg4 { [%eval 0,77/41][%emt 16]} 43.Bf1 {[%emt 0]} Rxd4 { [%eval 0,84/26][%emt 1]} 44.cxd4 {[%emt 4]} Kg6 { [%eval 0,84/46][%emt 7]} 45.d5 {[%emt 4]} Bb4 { [%eval 1/43][%emt 8]} 46.g4 {[%emt 0]} h4 { [%eval 1/42][%emt 4]} 47.Kh3 {[%emt 17]} Kg5 { [%eval 1/57][%emt 0]} 48.g3 {[%emt 5]} hxg3 { [%eval 1/59][%emt 0]} 49.Kxg3 {[%emt 3]} Bd6+ { [%eval 1/54][%emt 0]} 50.Kf3 {[%emt 3]} Be7 { [%eval 1/50][%emt 5]} 51.Be2 {[%emt 30]} Bc5 { [%eval 1/53][%emt 0]} 52.Ke4 {[%emt 16]} Kf6 { [%eval 1,58/32][%emt 9]} 53.b3 {[%emt 2]} Bd6 { [%eval 2,06/28][%emt 4]} 54.Kd4 {[%emt 29]} Ke7 { [%eval 5,58/30][%emt 1]} 55.g5 {[%emt 8]} Bf4 { [%eval 7,22/28][%emt 0]} 56.g6 {[%emt 24]} Kf6 { [%eval 14,56/27][%emt 0]} 57.Bd3 {[%emt 4]} f1=R { [%eval 17,64/24][%emt 2]} 58.Bxf1 {[%emt 4]} Kxg6 { [%eval 21,32/25][%emt 2]} 59.b4 {[%emt 0]} Bd2 { [%eval 59,29/22][%emt 6]} 60.bxa5 {[%emt 5]} Bxa5 { [%eval 128,41/32][%emt 3]} 61.Kc4 {[%emt 3]} Kf6 { [%eval 128,43/34][%emt 3]} 62.Kb5 {[%emt 1]} Bc7 { [%eval 128,44/43][%emt 1]} 63.a5 {[%emt 1]} Bh2 { [%eval 128,45/44][%emt 0]} 64.a6 {[%emt 1]} Bg1 { [%eval 128,46/45][%emt 0]} 65.d6 {[%emt 1]} Bf2 { [%eval 128,47/47][%emt 3]} 66.Kc6 {[%emt 0]} Kf7 { [%eval 128,47/48][%emt 0]} 67.d7 {[%emt 0]} Ke7 { [%eval 128,48/54][%emt 2]} 68.Kc7 {[%emt 0]} Kf7 { [%eval 128,49/44][%emt 2]} 69.d8=Q {[%emt 1]} Bb6+ { [%eval 128,50/70][%emt 3]} 70.Kxb6 {[%emt 1]} Kg6 { [%eval #5/127][%emt 2]} 71.a7 {[%emt 0]} Kf5 { [%eval #4/92][%emt 1]} 72.a8=Q {[%emt 0]} Kg6 { [%eval #3/127][%emt 1]} 73.Bd3+ {[%emt 0]} Kf7 { [%eval #2/127][%emt 1]} 74.Qf3+ {[%emt 0]} Ke6 { [%eval #1/127][%emt 1]} 75.Qfd5# {Mated[%emt 0]} 1-0[/pgn]
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

Post Reply