hi, wish you all a happy new year 2018
Inspired by the tests of Andreas Strangmueller
http://www.talkchess.com/forum/viewtopi ... 18&t=66180
I was starting a test of 1000 games, using the HERT test set.
Timer was 10min+10sec, which is average of 38 min per game.
Here is the result on 1 cpu each engine
And now, instead of doubling the time, I am raising the number of threads per engine to 4 cpu and will run the test with HERT set again.
regards, Clemens Keck
Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
Moderators: hgm, Rebel, chrisw
-
- Posts: 782
- Joined: Tue Dec 01, 2009 11:10 am
-
- Posts: 3
- Joined: Wed Apr 01, 2015 6:55 pm
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
can you do a live broadcast?
-
- Posts: 8
- Joined: Mon Dec 23, 2013 3:23 pm
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
Perhaps it's too early in thinking of the next of the next of the next, but I think that doubling again at least a couple of times till 16 cpu (T4?) is worth a 15-days test, in order to let possible some extrapolation for correspondence players.
Happy 2018!
Happy 2018!
-
- Posts: 61
- Joined: Wed Nov 20, 2013 10:42 am
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
Hi Clemens,
thanks for the tests, can you please share the pgn?
thanks for the tests, can you please share the pgn?
-
- Posts: 1525
- Joined: Tue Jul 15, 2014 12:47 pm
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
I see a very serious mistake in this match. You're using a debut book.Hugo wrote:hi, wish you all a happy new year 2018
Inspired by the tests of Andreas Strangmueller
http://www.talkchess.com/forum/viewtopi ... 18&t=66180
I was starting a test of 1000 games, using the HERT test set.
Timer was 10min+10sec, which is average of 38 min per game.
Here is the result on 1 cpu each engine
And now, instead of doubling the time, I am raising the number of threads per engine to 4 cpu and will run the test with HERT set again.
regards, Clemens Keck
-
- Posts: 782
- Joined: Tue Dec 01, 2009 11:10 am
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
I am ok with the HERT setKrzysztof Grzelak wrote:
I see a very serious mistake in this match. You're using a debut book.
The 500 Openings are played with reversed colors. ECO spread seems to be not bad at all.
Code: Select all
Games : 1000 (finished)
White Wins : 174 (17.4 %)
Black Wins : 68 ( 6.8 %)
Draws : 758 (75.8 %)
Unfinished : 0
White Perf. : 55.3 %
Black Perf. : 44.7 %
ECO A = 79 Games ( 7.9 %)
ECO B = 283 Games (28.3 %)
ECO C = 305 Games (30.5 %)
ECO D = 194 Games (19.4 %)
ECO E = 139 Games (13.9 %)
-
- Posts: 782
- Joined: Tue Dec 01, 2009 11:10 am
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
standings after 820 games
-
- Posts: 782
- Joined: Tue Dec 01, 2009 11:10 am
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
Hi All
the second testrun (4cpu each engine) is finnished.
How to interpret the result comparing T1 and T4 ?
Looks like Komodo with parallel search could reduce the distance to Stockfish by 13 Elo.
Regards, C.K.
the second testrun (4cpu each engine) is finnished.
How to interpret the result comparing T1 and T4 ?
Looks like Komodo with parallel search could reduce the distance to Stockfish by 13 Elo.
Regards, C.K.
Code: Select all
Games : 1000 (finished)
White Wins : 178 (17.8 %)
Black Wins : 50 ( 5.0 %)
Draws : 772 (77.2 %)
Unfinished : 0
White Perf. : 56.4 %
Black Perf. : 43.6 %
ECO A = 82 Games ( 8.2 %)
ECO B = 283 Games (28.3 %)
ECO C = 304 Games (30.4 %)
ECO D = 192 Games (19.2 %)
ECO E = 139 Games (13.9 %)
Program Elo + - Games Score Av.Op. Draws
1 Stockfish 261217 64 POPCNT T4 : 3017 10 10 1000 54.8 % 2983 77.2 %
2 Komodo 1973.00 64-bit T4 : 2983 10 10 1000 45.2 % 3017 77.2 %
-
- Posts: 8
- Joined: Mon Dec 23, 2013 3:23 pm
Re: Komodo 1973.00 64-bit - Stockfish 261217 64, match1000
Such a "reduction of distance" looks just "a bit beyond the expected", since it should be about 10, interpolating your conditions from the "normal reductions" in CEGT and CCRL (it's quite normal that increasing TC or #CPU the elo-gain should be diminishing, both in engineA vs engineB and in versionA1 vs versionA2).Hugo wrote:How to interpret the result comparing T1 and T4 ?
Looks like Komodo with parallel search could reduce the distance to Stockfish by 13 Elo.
CEGT 40/20
Stockfish 8 vs Stockfish 7 gains. 1CPU: 92(=3332-3240). 4CPU: 76(=3415-3339) -> reduction=16
Komodo 11.2 vs Komodo 10.4 gains. 1CPU: 13(=3328-3315). 4CPU: 24(=3422-3398) -> reduction=-11
Stockfish 8 vs Komodo 11.2 gains. 1CPU: 4(=3332-3328). 4CPU: -7(=3415-3422) -> reduction=11
CEGT 40/4
Stockfish 8 vs Stockfish 7 gains. 1CPU: 82(=3331-3249). 4CPU: 49(=3418-3369) -> reduction=33
Komodo 11.2 vs Komodo 10.4 gains. 1CPU: 26(=3330-3304). 4CPU: 7(=3404*-3397) -> reduction=19
Stockfish 8 vs Komodo 11.2 gains. 1CPU: 1(=3331-3330). 4CPU: 14(=3418-3404*) -> reduction=-13
*=Komodo 11.01
CCRL 40/4
Stockfish 8 vs Stockfish 7 gains. 1CPU: 67(=3423-3356). 4CPU: 73(=3495-3422) -> reduction=-6
Komodo 11.2 vs Komodo 10.4 gains. 1CPU: 58(=3426-3368). 4CPU: 43(=3514-3471) -> reduction=15
Stockfish 8 vs Komodo 11.2 gains. 1CPU: -3(=3423-3426). 4CPU: -19(=3495-3514) -> reduction=-16
Also, looking into http://chess.ultimaiq.net/scalability.htm it suggests a ~7 (or at least a <10) elo average reduction (although the abruptly low 1.7 elo reduction from the last two "doubling TC elo gains").
All these interpolations have to be quite rough, also because a few thousand games are not enough for them, but if this "bit beyond the expected" was confirmed by other similar tests (T16?) we could have a more definite clue that Komodo would work better than Stockfish at correspondence time controls (or at least that it would "gain" better at TCEC time controls).