Page 1 of 4
15,0000 game Komodo 11.01 test results
Posted: Thu May 25, 2017 3:11 pm
by PaulieD
15000 GAME GAUNTLET
i7 6700HQ | 4 @ 2.6-3.5 | Win10x64-16GB |Gui~CuteChess 0.9.4
Book~TopGM_2move.pgn | Hash~8MB | Cores~1 | HT~On
Concurrency 7 | Ponder~No | TB~No | Time~20"+.2" (~66 secs)
Latest ASM +132 Elo at this time control
Code: Select all
# PLAYER : RATING ERROR PLAYED (%) W D L D(%)
1 SF 170522ASMPD : 3266 5 3000 67.8 1396 1278 326 42.6
2 Komodo 11.01 : 3134 5 3000 32.2 326 1278 1396 42.6
Latest SF Dev +94 Elo at this time control
Code: Select all
# PLAYER : RATING ERROR PLAYED (%) W D L D(%)
1 SF 170523 : 3247 5 3000 63.0 1213 1354 433 45.1
2 Komodo 11.01 : 3153 5 3000 37.0 433 1354 1213 45.1
Seems SF8 is +80 Elo in 1 minute games.
Code: Select all
# PLAYER : RATING ERROR PLAYED (%) W D L D(%)
1 Stockfish 8 : 3240 4 3000 61.2 1100 1473 427 49.1
2 Komodo 11.01 : 3160 4 3000 38.8 427 1473 1100 49.1
H5.01 +56 Elo at this time control
Code: Select all
# PLAYER : RATING ERROR PLAYED (%) W D L D(%)
1 Houdini 5.01 Pro : 3228 4 3000 57.7 1006 1449 545 48.3
2 Komodo 11.01 : 3172 4 3000 42.3 545 1449 1006 48.3
+10 Elo over K10.4
Code: Select all
# PLAYER : RATING ERROR PLAYED (%) W D L D(%)
1 Komodo 11.01 : 3205 4 3000 51.3 679 1722 599 57.4
2 Komodo 10.4 : 3195 4 3000 48.7 599 1722 679 57.4
Re: 15,0000 game Komodo 11.01 test results
Posted: Thu May 25, 2017 8:43 pm
by yanquis1972
the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
Re: 15,0000 game Komodo 11.01 test results
Posted: Thu May 25, 2017 9:36 pm
by mjlef
yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.
Mark
Re: 15,0000 game Komodo 11.01 test results
Posted: Thu May 25, 2017 9:45 pm
by PaulieD
mjlef wrote:yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.
Mark
This time control is not 20 seconds. It is the mid level time control used on the Stockfish Testing Framework. 10 + .1 is STC, 20 + .2 is MTC, and 60 + .6 is LTC.
All engine settings were at default.
Using the additional time increment .2 lengthens the games to about 66 seconds each.
That is not a "super fast" time control on todays hardware.
Re: 15,0000 game Komodo 11.01 test results
Posted: Thu May 25, 2017 11:56 pm
by David Xu
PaulieD wrote:mjlef wrote:yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
We quote results for times like 2 mins + 1 sec, now 20 seconds. The super fast time control is know to hurt Komodo more, but this does seem very strange. Perhaps some setting? I see hyperthreading was on, so the 20 second level is more like 10 or 11 second games on a non-hyperthreading core.
Mark
This time control is not 20 seconds. It is the mid level time control used on the Stockfish Testing Framework. 10 + .1 is STC, 20 + .2 is MTC, and 60 + .6 is LTC.
All engine settings were at default.
Using the additional time increment .2 lengthens the games to about 66 seconds each.
That is not a "super fast" time control on todays hardware.
It depends on what you're comparing the time control to. Perhaps in SF testing, 20 + .2 testing counts as MTC, but most rating lists don't test at such short TC's. (Nor, for that matter, are most engine games played at such short TC's
outside of testing.)
Also, the default settings of Komodo include Contempt being set at 10, which is known to hurt against roughly equal opponents such as Houdini and Stockfish. So yeah, that might contribute toward explaining this result. (Mark, Larry, if you're reading this, I would definitely recommend setting the default Contempt to 0 in future releases of Komodo, just to avoid stuff like this happening.)
Re: 15,0000 game Komodo 11.01 test results
Posted: Fri May 26, 2017 1:59 am
by lkaufman
I have run many thousands of fast games using LittleBlitzer in Windows at time limits like 1' + 1", 2' + 1", and 5' + 1", between Stockfish 8 and Komodo 11.01, and between Houdini 5.01 and Komodo 11.01, all single thread tests with hyperthreading on on 20 and 24 core machines, concurrency 32 and 48 respectively. These are using defaults, i.e. Contempt = 10 (probably with Contempt zero they would be a few elo better). The results have been remarkably consistent. We lose to SF8 by about 32 elo, and we barely beat Houdini (average plus 1 elo). I think that the amount of increment is the main factor in our results against these engines; the larger it is, the better we do. Base time doesn't seem to matter much. This would explain why the results in this thread are so poor with 0.2" inc. or even zero inc. in the case of 1 min results. Seems like one second increment is about the minimum to have meaningful results. I think this may also be why Komodo has done relatively poorly in games at 40 min plus 4 sec; it's still blitz with a 4 sec increment. I'm not sure why the increment seems so important for Komodo to shine. I'd like to see some tests with the FIDE standard 30" inc. (i.e. 90 min plus 30 sec). I note that Komodo always does well in CCRL and CEGT testing, which is more like testing with a large increment than a small one.
Re: 15,0000 game Komodo 11.01 test results
Posted: Fri May 26, 2017 2:18 am
by Gusev
Too many time forfeits at such short TCs?
yanquis1972 wrote:the +10elo over K10.4 seems applicable based on IPON & larry/mark's estimates, the rest...not so much. not sure why that is though, any ideas?
Re: 15,0000 game Komodo 11.01 test results
Posted: Fri May 26, 2017 4:39 am
by PaulieD
You will get time forfeits with Little Blitzer, but not one with Cute Chess Gui 0.9.4.
There was nothing wrong with the settings/conditions. All games were with 20 +.2 for 66 secs. I used the term one minute in one set of games looslely.
They are duplicateable with the same conditions/settings.
Komodo may need a lot more time and cores than this time control...a lot more.
Re: 15,0000 game Komodo 11.01 test results
Posted: Fri May 26, 2017 5:40 am
by shrapnel
David Xu wrote: (Mark, Larry, if you're reading this, I would definitely recommend setting the default Contempt to 0 in future releases of Komodo, just to avoid stuff like this happening.)
Mark and Larry may or may not have got the Contempt Setting right, but they CERTAINLY got the Time Management settings spot on !
Yesterday I played 2 consecutive 5'+0 games on Infinity Chess with 2 different opponents, while using Komodo 11.01.
In both games, end-game Position was about equal, with 0.00 score.
Only difference was that Komodo had used up much less time than my opponents who were using Stockfish/Brainfish.
In fact, Komodo seemed to be aware in both games that Stockfish/Brainfish was in time trouble !
Komodo simply played faster and faster, without falling into the Trap of "Threefold-Repetition".
In one game, I remember, it simply moved a Bishop back and forth very fast, each time to a different square !
Needless to say, Stockfish/Brainfish Timed Out in both games!
So, maybe not a fair win, but as they say in chess and war, a win is a win, no matter HOW it is achieved.
Re: 15,0000 game Komodo 11.01 test results
Posted: Fri May 26, 2017 1:36 pm
by shrapnel
A game Komodo won against the over-rated asmfish.....
[Event "Blitz 4' + 2'', Rated"]
[Site ""]
[Date "2017.5.26"]
[Round ""]
[White "shrapnel"]
[Black "ocirema"]
[Result "1-0"]
[ECO "C53"]
[WhiteElo "2472"]
[BlackElo "2612"]
[WhiteEngine "komodo-11.01-64bit_CTG"]
[BlackEngine "asmFishW_2017-05-22_popcnt"]
[EventType ""]
[WhiteTeam ""]
[BlackTeam ""]
[PlyCount "149"]
[TimeControl "240+2"]
[pgn]1.e4 {[%emt 0]} e5 {[%emt 1][%B]} 2.Nf3 {[%emt 0]} Nc6 {[%emt 0][%B]} 3.Bc4 {[%emt 0]} Bc5 {[%emt 0][%B]} 4.c3 {[%emt 0]} Nf6 {[%emt 0][%B]} 5.d3 {[%emt 0]} d6 {[%emt 0][%B]} 6.O-O {[%emt 0]} a5 {[%emt 1][%B]} 7.Bg5 {[%emt 0]} O-O {[%emt 1][%B]} 8.a4 {[%emt 0]} h6 {[%emt 1][%B]} 9.Bh4 {[%emt 0]} g5 {[%emt 1][%B]} 10.Bg3 {[%emt 0]} Kg7 {[%emt 1][%B]} 11.Nbd2 {[%emt 0]} g4 {[%emt 0]} 12.Nh4 {[%emt 0]} Nh5 { [%eval 0,02/17][%emt 0]} 13.Bb5 {[%emt 0]} Qg5 { [%eval 0,13/18][%emt 0]} 14.Nc4 {[%emt 0]} Nxg3 { [%eval 0,27/19][%emt 0]} 15.hxg3 {[%emt 0]} Ne7 { [%eval 0,27/19][%emt 0]} 16.d4 {[%emt 0]} Ba7 { [%eval 0,25/18][%emt 0]} 17.Ne3 {[%emt 0]} f5 { [%eval 0,38/18][%emt 1]} 18.exf5 {[%emt 0]} Nxf5 { [%eval 0,44/16][%emt 0]} 19.Nexf5+ {[%emt 0]} Bxf5 { [%eval -0,03/20][%emt 0]} 20.Be2 {[%emt 0]} h5 { [%eval 0,25/18][%emt 0]} 21.dxe5 {[%emt 0]} Bh7 { [%eval 0,10/19][%emt 0]} 22.exd6 {[%emt 0]} cxd6 { [%eval 0,15/20][%emt 0]} 23.Kh2 {[%emt 0]} Rxf2 { [%eval 0,27/20][%emt 0]} 24.Rxf2 {[%emt 0]} Bxf2 { [%eval 0,13/21][%emt 0]} 25.Qxd6 {[%emt 0]} Re8 { [%eval 0,09/24][%emt 0]} 26.Bc4 {[%emt 0]} Qe5 { [%eval 0,08/25][%emt 0]} 27.Qxe5+ {[%emt 0]} Rxe5 { [%eval 0.00/23][%emt 0]} 28.Rf1 {[%emt 0]} Be1 { [%eval 0,08/23][%emt 0]} 29.Rf7+ {[%emt 0]} Kh6 { [%eval 0,08/24][%emt 0]} 30.Rxb7 {[%emt 1]} Re3 { [%eval 0,08/27][%emt 0]} 31.Rb6+ {[%emt 4]} Kg5 { [%eval 1/45][%emt 2:02]} 32.Rb5+ {[%emt 6]} Kf6 { [%eval 1/45][%emt 11]} 33.Rb6+ {[%emt 0]} Ke7 { [%eval 1/45][%emt 1:11]} 34.Rb7+ {[%emt 6]} Kf6 { [%eval 1/43][%emt 15]} 35.Nf3 {[%emt 0]} gxf3 { [%eval 0,85/31][%emt 12]} 36.Rxh7 {[%emt 0]} f2 { [%eval 0,91/33][%emt 6]} 37.Rf7+ {[%emt 18]} Kg5 { [%eval 0,98/33][%emt 4]} 38.Ra7 {[%emt 6]} Re5 { [%eval 1/32][%emt 0]} 39.Ra8 {[%emt 10]} Kh6 { [%eval 0,92/36][%emt 25]} 40.Rd8 {[%emt 26]} Re4 { [%eval 0,37/30][%emt 9]} 41.Rd6+ {[%emt 5]} Kg7 { [%eval 0,57/31][%emt 0]} 42.Rd4 {[%emt 3]} Rg4 { [%eval 0,77/41][%emt 16]} 43.Bf1 {[%emt 0]} Rxd4 { [%eval 0,84/26][%emt 1]} 44.cxd4 {[%emt 4]} Kg6 { [%eval 0,84/46][%emt 7]} 45.d5 {[%emt 4]} Bb4 { [%eval 1/43][%emt 8]} 46.g4 {[%emt 0]} h4 { [%eval 1/42][%emt 4]} 47.Kh3 {[%emt 17]} Kg5 { [%eval 1/57][%emt 0]} 48.g3 {[%emt 5]} hxg3 { [%eval 1/59][%emt 0]} 49.Kxg3 {[%emt 3]} Bd6+ { [%eval 1/54][%emt 0]} 50.Kf3 {[%emt 3]} Be7 { [%eval 1/50][%emt 5]} 51.Be2 {[%emt 30]} Bc5 { [%eval 1/53][%emt 0]} 52.Ke4 {[%emt 16]} Kf6 { [%eval 1,58/32][%emt 9]} 53.b3 {[%emt 2]} Bd6 { [%eval 2,06/28][%emt 4]} 54.Kd4 {[%emt 29]} Ke7 { [%eval 5,58/30][%emt 1]} 55.g5 {[%emt 8]} Bf4 { [%eval 7,22/28][%emt 0]} 56.g6 {[%emt 24]} Kf6 { [%eval 14,56/27][%emt 0]} 57.Bd3 {[%emt 4]} f1=R { [%eval 17,64/24][%emt 2]} 58.Bxf1 {[%emt 4]} Kxg6 { [%eval 21,32/25][%emt 2]} 59.b4 {[%emt 0]} Bd2 { [%eval 59,29/22][%emt 6]} 60.bxa5 {[%emt 5]} Bxa5 { [%eval 128,41/32][%emt 3]} 61.Kc4 {[%emt 3]} Kf6 { [%eval 128,43/34][%emt 3]} 62.Kb5 {[%emt 1]} Bc7 { [%eval 128,44/43][%emt 1]} 63.a5 {[%emt 1]} Bh2 { [%eval 128,45/44][%emt 0]} 64.a6 {[%emt 1]} Bg1 { [%eval 128,46/45][%emt 0]} 65.d6 {[%emt 1]} Bf2 { [%eval 128,47/47][%emt 3]} 66.Kc6 {[%emt 0]} Kf7 { [%eval 128,47/48][%emt 0]} 67.d7 {[%emt 0]} Ke7 { [%eval 128,48/54][%emt 2]} 68.Kc7 {[%emt 0]} Kf7 { [%eval 128,49/44][%emt 2]} 69.d8=Q {[%emt 1]} Bb6+ { [%eval 128,50/70][%emt 3]} 70.Kxb6 {[%emt 1]} Kg6 { [%eval #5/127][%emt 2]} 71.a7 {[%emt 0]} Kf5 { [%eval #4/92][%emt 1]} 72.a8=Q {[%emt 0]} Kg6 { [%eval #3/127][%emt 1]} 73.Bd3+ {[%emt 0]} Kf7 { [%eval #2/127][%emt 1]} 74.Qf3+ {[%emt 0]} Ke6 { [%eval #1/127][%emt 1]} 75.Qfd5# {Mated[%emt 0]} 1-0[/pgn]