Testing Komodo9: 1440 Games
Moderators: hgm, Rebel, chrisw
-
lkaufman
- Posts: 5961
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: Testing Komodo 9: 1440 games.
Going to pure one minute level should be compared to pure 2' and pure 4' level, since the results show pretty clearly that Komodo does better with increment than without, given roughly equal total time per game. My guess is that we'll drop another 8 elo or so compared with pure 2'. I recall that a couple years back I estimated that each doubling of the time limit increases our results vs. Houdini by about 8 elo, just the number you found against all three opponents. I'm not at all surprised that at time limits like game/10" or so we are only at Houdini 4 level. For one thing, results at this level are heavily dependent on Move Overhead settings; they don't tell very much about actual engine strength. Such fast results are useful for engine development, where overhead is held constant, but not for comparing unrelated engines. But one minute is probably enough time to minimize this overhead issue.
Komodo rules!
-
Tomcass
- Posts: 786
- Joined: Sun Apr 16, 2006 9:09 pm
Re: Testing Komodo 9: 1440 games.
Thanks for your comment Larry. I obviously agree that the comparison among my tests must be consistent in terms of time control.
When I'll finish this 1 minute test I will see what happen at 16 Billion nodes -average per game- time control. (8 + 0) and (4 + 4). It will be a quite long, long, long test. Perhaps we can expect an improvement of around 7 or 8 elo points in relation to the curent record in my tests by Komodo 9.1 (3247 ELO)?.
Regards,
Tom.
When I'll finish this 1 minute test I will see what happen at 16 Billion nodes -average per game- time control. (8 + 0) and (4 + 4). It will be a quite long, long, long test. Perhaps we can expect an improvement of around 7 or 8 elo points in relation to the curent record in my tests by Komodo 9.1 (3247 ELO)?.
Regards,
Tom.
-
Tomcass
- Posts: 786
- Joined: Sun Apr 16, 2006 9:09 pm
Re: Testing Komodo 9: 1440 games.
TESTING KOMODO 9.1 = 1440 GAMES.
This time I have reduced to 25% of my standard test parameters the number of average nodes per game -from 8 to 2 Billion-. (Nodes per game: Average seconds per game x average nodes per second.)
Pleaso note that this test is only Fixed Time control –no incremental- so that we must compare it with Fixed Time Control results in the two previous tests. Let’s see!.
6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177
Time Control 1 + 0
Komodo 9.1 64-bit_x6 - SF050615MZ 64 POPCNT_6 114.5 - 125.5 +45/=139/-56 47.71%
Komodo 9.1 64-bit_x6 - Houdini 4 x64_st_X6_CT0 153.0 - 87.0 +91/=124/-25 63.75%
Komodo 9.1 64-bit_x6 - Gull 3 x64 XP 169.0 - 71.0 +113/=112/-15 70.42%
Total Score using 6 Cores: 436.5 – 283.5 = 60.62%
720 Games=
http://www.mediafire.com/download/axuuz ... 5%_NPS.pgn
8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 44.92
Knodes per Second: 21.562
Time Control 1 + 0
Komodo 9.1 64-bit_x8 - SF050615MZ 64 AVX_8 116.0 - 124.0 +32/=168/-40 48.33%
Komodo 9.1 64-bit_x8 - Houdini 4 Pro x64_Ct0_8 152.5 - 87.5 +83/=139/-18 63.54%
Komodo 9.1 64-bit_x8 - Gull 3 x64 XPx8 173.5 - 66.5 +123/=101/-16 72.29%
720 Games=
http://www.mediafire.com/download/tj2x4 ... 25%BNG.pgn
Total score using 8 Cores= 442.0 – 278.0= 61.87%
Global Score after 1440 games= 878.5 – 561.5= 61.01%
Against Stockfish 050615 MZ (3238) = 48.02% Houdini 4 (3136)= 63.65% Gull (3103)= 71.35%
Average Elo of Opponents= 3159
Estimated Elo Performance for Komodo 9.1 at 2 Billion nodes per game = 3136
Error bars: 13 Elo points.
And comparting the sequence of scores at Fixed Time Control fot Komodo 9.1:
At 4 minutes + 0 = 61.25% . Estimated Elo = 3138 after 720 games
At 2 minutes + 0 = 60.76%. Estimated Elo = 3134 after 720 games
At 1 minute + 0 = 61.01%. Estimated Elo = 3136 after 1440 games
The difference is not relevant at statistical effects.
And comparing the sequence of scores for Komodo 9.1 at Incremental Time control:
At 2 minutes + 2 seconds = 63.75%. Estimated Elo = 3155 after 720 games
At 1 minute + 1 second = 62.15%. Estimated Elo= 3244 after 720 games.
Unfortunately I have not been able to test Komodo 9.1 at 30 seconds + 0.5 second time control. I have not discovered how to do it in my Fritz 14 GUI. If it is possible, can somebody tell me how to do it please?. Thanks!.
Komodo 9.1 seems substantially stronger at incremental time control -around 13 or 14 Elo Points- than at fixed TC, as Larry Kauffman said. Before drawing any other consequence of these tests I will start the latest leg of this serie testing Komodo 9.1 at 8 +0 and 4 +4 time control. Roughly 16 Billion nodes per game on average. I will reduce the number of games to 720.
Best regards from Barcelona.
Tom.
This time I have reduced to 25% of my standard test parameters the number of average nodes per game -from 8 to 2 Billion-. (Nodes per game: Average seconds per game x average nodes per second.)
Pleaso note that this test is only Fixed Time control –no incremental- so that we must compare it with Fixed Time Control results in the two previous tests. Let’s see!.
6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177
Time Control 1 + 0
Komodo 9.1 64-bit_x6 - SF050615MZ 64 POPCNT_6 114.5 - 125.5 +45/=139/-56 47.71%
Komodo 9.1 64-bit_x6 - Houdini 4 x64_st_X6_CT0 153.0 - 87.0 +91/=124/-25 63.75%
Komodo 9.1 64-bit_x6 - Gull 3 x64 XP 169.0 - 71.0 +113/=112/-15 70.42%
Total Score using 6 Cores: 436.5 – 283.5 = 60.62%
720 Games=
http://www.mediafire.com/download/axuuz ... 5%_NPS.pgn
8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 44.92
Knodes per Second: 21.562
Time Control 1 + 0
Komodo 9.1 64-bit_x8 - SF050615MZ 64 AVX_8 116.0 - 124.0 +32/=168/-40 48.33%
Komodo 9.1 64-bit_x8 - Houdini 4 Pro x64_Ct0_8 152.5 - 87.5 +83/=139/-18 63.54%
Komodo 9.1 64-bit_x8 - Gull 3 x64 XPx8 173.5 - 66.5 +123/=101/-16 72.29%
720 Games=
http://www.mediafire.com/download/tj2x4 ... 25%BNG.pgn
Total score using 8 Cores= 442.0 – 278.0= 61.87%
Global Score after 1440 games= 878.5 – 561.5= 61.01%
Against Stockfish 050615 MZ (3238) = 48.02% Houdini 4 (3136)= 63.65% Gull (3103)= 71.35%
Average Elo of Opponents= 3159
Estimated Elo Performance for Komodo 9.1 at 2 Billion nodes per game = 3136
Error bars: 13 Elo points.
And comparting the sequence of scores at Fixed Time Control fot Komodo 9.1:
At 4 minutes + 0 = 61.25% . Estimated Elo = 3138 after 720 games
At 2 minutes + 0 = 60.76%. Estimated Elo = 3134 after 720 games
At 1 minute + 0 = 61.01%. Estimated Elo = 3136 after 1440 games
The difference is not relevant at statistical effects.
And comparing the sequence of scores for Komodo 9.1 at Incremental Time control:
At 2 minutes + 2 seconds = 63.75%. Estimated Elo = 3155 after 720 games
At 1 minute + 1 second = 62.15%. Estimated Elo= 3244 after 720 games.
Unfortunately I have not been able to test Komodo 9.1 at 30 seconds + 0.5 second time control. I have not discovered how to do it in my Fritz 14 GUI. If it is possible, can somebody tell me how to do it please?. Thanks!.
Komodo 9.1 seems substantially stronger at incremental time control -around 13 or 14 Elo Points- than at fixed TC, as Larry Kauffman said. Before drawing any other consequence of these tests I will start the latest leg of this serie testing Komodo 9.1 at 8 +0 and 4 +4 time control. Roughly 16 Billion nodes per game on average. I will reduce the number of games to 720.
Best regards from Barcelona.
Tom.
-
JJJ
- Posts: 1346
- Joined: Sat Apr 19, 2014 1:47 pm
Re: Testing Komodo 9: 1440 games.
Knowing that Komodo score better with time increment, will you test gull and houdini too ?
Also, will you remove time control without increment for your next test of engine ? I think slow time increment is more accurate for knowing the strenght of an engine than without. Because when you make analysis, you won't reduce the time you use. Time increment is the garanty to always get a minimum of thinking time.
Sure Stockfish is the best without, but strenght is always reduced.
Also, will you remove time control without increment for your next test of engine ? I think slow time increment is more accurate for knowing the strenght of an engine than without. Because when you make analysis, you won't reduce the time you use. Time increment is the garanty to always get a minimum of thinking time.
Sure Stockfish is the best without, but strenght is always reduced.
-
Tomcass
- Posts: 786
- Joined: Sun Apr 16, 2006 9:09 pm
Re: Testing Komodo 9: 1440 games.
Hi, Jean Baptiste.
Thanks for your interest.
My plan is to run this latest test with Komodo 9.1 and offer you the results so that everyone can draw his/her owwn conclusions. This test is dual, as usual, where 360 games will be played at 8 min + 0 and the other 360 at 4 minutes + 4 seconds incremental.
Then I will go back to Stockfish Development.
Kind regards,
Tom.
Thanks for your interest.
My plan is to run this latest test with Komodo 9.1 and offer you the results so that everyone can draw his/her owwn conclusions. This test is dual, as usual, where 360 games will be played at 8 min + 0 and the other 360 at 4 minutes + 4 seconds incremental.
Then I will go back to Stockfish Development.
Kind regards,
Tom.
-
JJJ
- Posts: 1346
- Joined: Sat Apr 19, 2014 1:47 pm
Re: Testing Komodo 9: 1440 games.
So you wont change how you test Stockfish ? 
-
Tomcass
- Posts: 786
- Joined: Sun Apr 16, 2006 9:09 pm
Re: Testing Komodo 9: 1440 games.
I decided my testing methodology some years ago, Jean Baptiste. I'll keep it (50% fixed and 50% incremental time control) to get the maximum level of internal consistency and robustness in my tests.
Anyway ... I am always open to listen any intelligent suggestions.
Tom.
... by the way, the first half of my latest -by now!- test with Komodo 9.1 will be finished tonight. I will post it tomorrow morning. (360 games, 180 fixed and 180 incremental T.C.)
Anyway ... I am always open to listen any intelligent suggestions.
Tom.
... by the way, the first half of my latest -by now!- test with Komodo 9.1 will be finished tonight. I will post it tomorrow morning. (360 games, 180 fixed and 180 incremental T.C.)
-
Tomcass
- Posts: 786
- Joined: Sun Apr 16, 2006 9:09 pm
Re: Testing Komodo 9: 1440 games.
Wooops!
Sorry, I have seen a typo in my post above with the score of Komodo 9.1. Since I can not edit it I copy now the right calculation of the Estimated Elo:
Global Score after 1440 games= 878.5 – 561.5= 61.01%
Against Stockfish 050615 MZ (3238) = 48.02% Houdini 4 (3136)= 63.65% Gull (3103)= 71.35%
Average Elo of Opponents= 3159
Estimated Elo Performance for Komodo 9.1 at 2 Billion nodes per game = 3236
Error bars: 13 Elo points.
And comparting the sequence of scores at Fixed Time Control fot Komodo 9.1:
At 4 minutes + 0 = 61.25% . Estimated Elo = 3238 after 720 games
At 2 minutes + 0 = 60.76%. Estimated Elo = 3234 after 720 games
At 1 minute + 0 = 61.01%. Estimated Elo = 3236 after 1440 games
The difference is not relevant at statistical effects.
And comparing the sequence of scores for Komodo 9.1 at Incremental Time control:
At 2 minutes + 2 seconds = 63.75%. Estimated Elo = 3255 after 720 games
At 1 minute + 1 second = 62.15%. Estimated Elo= 3244 after 720 games.
Tom.
Sorry, I have seen a typo in my post above with the score of Komodo 9.1. Since I can not edit it I copy now the right calculation of the Estimated Elo:
Global Score after 1440 games= 878.5 – 561.5= 61.01%
Against Stockfish 050615 MZ (3238) = 48.02% Houdini 4 (3136)= 63.65% Gull (3103)= 71.35%
Average Elo of Opponents= 3159
Estimated Elo Performance for Komodo 9.1 at 2 Billion nodes per game = 3236
Error bars: 13 Elo points.
And comparting the sequence of scores at Fixed Time Control fot Komodo 9.1:
At 4 minutes + 0 = 61.25% . Estimated Elo = 3238 after 720 games
At 2 minutes + 0 = 60.76%. Estimated Elo = 3234 after 720 games
At 1 minute + 0 = 61.01%. Estimated Elo = 3236 after 1440 games
The difference is not relevant at statistical effects.
And comparing the sequence of scores for Komodo 9.1 at Incremental Time control:
At 2 minutes + 2 seconds = 63.75%. Estimated Elo = 3255 after 720 games
At 1 minute + 1 second = 62.15%. Estimated Elo= 3244 after 720 games.
Tom.
-
Tomcass
- Posts: 786
- Joined: Sun Apr 16, 2006 9:09 pm
Re: Testing Komodo 9: 1440 games.
TESTING KOMODO 9.1 = 720 GAMES. FIRST HALF: FIXED TIME CONTROL 8 MINUTES + 0 PER GAME: 360 GAMES.
Average= 16 Billion nodes per game.
6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177
Time Control 8 + 0
Komodo 9.1 64-bit_x6 - SF050615MZ 64 POPCNT_6 34.5 - 25.5 +17/=35/-8 57.50%
Komodo 9.1 64-bit_x6 - Houdini 4 x64_st_X6_CT0 36.5 - 23.5 +16/=41/-3 60.83%
Komodo 9.1 64-bit_x6 - Gull 3 x64 XP 38.5 - 21.5 +22/=33/-5 64.17%
180 Games=
http://www.mediafire.com/download/aa2u0 ... NPG_OK.pgn
8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 44.92
Knodes per Second: 21.562
Time Control 8 + 0
Komodo 9.1 64-bit_x8 - SF050615MZ 64 AVX_8 30.5 - 29.5 +9/=43/-8 50.83%
Komodo 9.1 64-bit_x8 - Houdini 4 Pro x64_Ct0_8 41.0 - 19.0 +24/=34/-2 68.33%
Komodo 9.1 64-bit_x8 - Gull 3 x64 XPx8 44.0 - 16.0 +29/=30/-1 73.33%
180 Games=
http://www.mediafire.com/download/ux8bx ... 0%_NPG.pgn
Score at 8 + 0 after 360 Games= 225.0 – 135.0 = 3246 Elo
And the whole sequence at fixed Time Control after 3.240 games is:
At 8 minutes + 0 = 62.50% Estimated Elo = 3246 after 360 games
At 4 minutes + 0 = 61.25% . Estimated Elo = 3238 after 720 games
At 2 minutes + 0 = 60.76%. Estimated Elo = 3234 after 720 games
At 1 minute + 0 = 61.01%. Estimated Elo = 3236 after 1440 games
I have just started the second half of this test at 4 minutes + 4 seconds Time Control.
Regards,
Tom.
Average= 16 Billion nodes per game.
6 real cores
Ponder: Off.
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 512
Relative Speed: 29.54
Knodes per second: 14.177
Time Control 8 + 0
Komodo 9.1 64-bit_x6 - SF050615MZ 64 POPCNT_6 34.5 - 25.5 +17/=35/-8 57.50%
Komodo 9.1 64-bit_x6 - Houdini 4 x64_st_X6_CT0 36.5 - 23.5 +16/=41/-3 60.83%
Komodo 9.1 64-bit_x6 - Gull 3 x64 XP 38.5 - 21.5 +22/=33/-5 64.17%
180 Games=
http://www.mediafire.com/download/aa2u0 ... NPG_OK.pgn
8 real cores
Ponder: Off
GUI: Fritz 14
Book: Perfect 2014c – Limit 5 moves. Sedat –
No tablebases. No RTB used.
Hash 1024
Relative Speed: 44.92
Knodes per Second: 21.562
Time Control 8 + 0
Komodo 9.1 64-bit_x8 - SF050615MZ 64 AVX_8 30.5 - 29.5 +9/=43/-8 50.83%
Komodo 9.1 64-bit_x8 - Houdini 4 Pro x64_Ct0_8 41.0 - 19.0 +24/=34/-2 68.33%
Komodo 9.1 64-bit_x8 - Gull 3 x64 XPx8 44.0 - 16.0 +29/=30/-1 73.33%
180 Games=
http://www.mediafire.com/download/ux8bx ... 0%_NPG.pgn
Score at 8 + 0 after 360 Games= 225.0 – 135.0 = 3246 Elo
And the whole sequence at fixed Time Control after 3.240 games is:
At 8 minutes + 0 = 62.50% Estimated Elo = 3246 after 360 games
At 4 minutes + 0 = 61.25% . Estimated Elo = 3238 after 720 games
At 2 minutes + 0 = 60.76%. Estimated Elo = 3234 after 720 games
At 1 minute + 0 = 61.01%. Estimated Elo = 3236 after 1440 games
I have just started the second half of this test at 4 minutes + 4 seconds Time Control.
Regards,
Tom.
-
egiovannotti
- Posts: 38
- Joined: Wed Oct 31, 2012 9:28 am
Re: Testing Komodo 9: 1440 games.
The links do not work.