Moderators: hgm, Rebel, chrisw
Which compile is best for Intel i7 processors (2nd generation or 3 generation)?Modern Times wrote:My understanding is that the only "official" versions are the ones here:
https://stockfishchess.org/
In terms of unofficial daily builds, take your pick. I get mine here:
http://chess.ultimaiq.net/stockfish.html
Sorry - for IDeA should UCI analysis mode be checked or unchecked?lkaufman wrote:Yes.Werewolf wrote:Larry, for use in IDeA, should contempt be set to zero?lkaufman wrote:I think we'll do ok relative to stockfish in tests like Frank's where each opponent plays against many other opponents, but we will not look so good in direct matches with Stockfish. The reasons for this are not all clear, although "Contempt" is a factor. It's up to each person to decide whether a direct match or a rr tournament is a better way to decide which of two engines is stronger. There are arguments both ways.S.Taylor wrote:What _I_ want to see is how this new komodo 10.2 does vs the SF that played in stage 3 of TCEC, OR, the version of SF that played in Graham Banks matches between sf and komodo.lkaufman wrote:
It is very frustrating to have two moving targets at the same time.
e.g. let's say komodo beats ASM, how will we know if the older SF version, which Graham used, wouldn't have crushed komodo 10.2 like it did to komodo 10.1?
We should first test a moving target with a standing target before we go on to the second moving target.
Both would use modern. 4th gen minimum for Haswell+.h1a8 wrote:Which compile is best for Intel i7 processors (2nd generation or 3 generation)?Modern Times wrote:My understanding is that the only "official" versions are the ones here:
https://stockfishchess.org/
In terms of unofficial daily builds, take your pick. I get mine here:
http://chess.ultimaiq.net/stockfish.html
Thanks
Well perhaps you did relatively OK in the past but nowadays... this was latest result by Frank. A clear win for SF 180916 against all, Komodo 10 and 10.1 includedlkaufman wrote:I think we'll do ok relative to stockfish in tests like Frank's where each opponent plays against many other opponents, but we will not look so good in direct matches with Stockfish. The reasons for this are not all clear, although "Contempt" is a factor. It's up to each person to decide whether a direct match or a rr tournament is a better way to decide which of two engines is stronger. There are arguments both ways.S.Taylor wrote:What _I_ want to see is how this new komodo 10.2 does vs the SF that played in stage 3 of TCEC, OR, the version of SF that played in Graham Banks matches between sf and komodo.lkaufman wrote:We have released komodo 10.2 at komodochess.com. We estimate based on our testing that it is about 22 elo stronger than komodo 10.1 and about 30 elo stronger than komodo 10 at three minutes plus one second increment, on one or many threads. This is the largest elo gain we have had between versions since going to the subscription model. There were so many changes that it is very difficult to say which ones should get most of the credit. Evaluation, search, and time management were all changed significantly. Features (UCI options) are the same as in komodo 10.1. As usual, it is free to subscribers, and available at a 20% discount to those who bought komodo 9 or a later version.
It is very frustrating to have two moving targets at the same time.
e.g. let's say komodo beats ASM, how will we know if the older SF version, which Graham used, wouldn't have crushed komodo 10.2 like it did to komodo 10.1?
We should first test a moving target with a standing target before we go on to the second moving target.
Code: Select all
1 SF 18Sep2016 BMI2 x64 C10 xxxxx 33.0 39.0 38.5 36.0 40.0 37.0 36.0 39.5 42.0 39.0 42.5 41.5 42.5 42.5
2 Komodo 10 x64 17.0 xxxxx 32.0 37.0 40.0 36.5 37.0 37.0 40.5 41.0 40.5 36.5 42.0 39.5 39.0
3 Houdini 4 STD B x64 11.0 18.0 xxxxx 26.5 26.0 27.5 32.5 29.5 30.5 34.0 32.0 33.0 28.5 32.0 38.0
4 Fire 4 x64 11.5 13.0 23.5 xxxxx 23.5 25.0 25.0 26.5 32.0 28.5 31.0 29.0 32.5 31.5 32.0
5 GullChess 3.0 BMI2 x64 14.0 10.0 24.0 26.5 xxxxx 31.0 23.5 24.5 28.0 30.5 30.0 32.0 26.5 31.5 32.5
6 Andscacs 0.872 BMI2 x64 10.0 13.5 22.5 25.0 19.0 xxxxx 26.0 25.0 26.0 23.5 26.5 31.5 25.0 28.5 31.0
7 Equinox 3.30 x64 13.0 13.0 17.5 25.0 26.5 24.0 xxxxx 24.0 23.5 26.0 29.0 27.5 28.5 32.5 30.5
8 Fizbo 1.8 BMI2 x64 14.0 13.0 20.5 23.5 25.5 25.0 26.0 xxxxx 29.5 23.5 25.5 27.0 28.0 30.5 29.0
9 Critter 1.6a x64 10.5 9.5 19.5 18.0 22.0 24.0 26.5 20.5 xxxxx 24.0 27.5 26.5 28.0 29.5 28.5
10 Fritz 15 x64 8.0 9.0 16.0 21.5 19.5 26.5 24.0 26.5 26.0 xxxxx 26.5 22.5 25.0 26.5 24.0
11 Nirvanachess 2.3 POP x64 11.0 9.5 18.0 19.0 20.0 23.5 21.0 24.5 22.5 23.5 xxxxx 26.5 27.0 28.5 22.5
12 Hannibal 1.7 x64 7.5 13.5 17.0 21.0 18.0 18.5 22.5 23.0 23.5 27.5 23.5 xxxxx 24.0 24.0 29.5
13 Chiron 3 x64 8.5 8.0 21.5 17.5 23.5 25.0 21.5 22.0 22.0 25.0 23.0 26.0 xxxxx 21.5 22.0
14 Protector 1.9.0 x64 7.5 10.5 18.0 18.5 18.5 21.5 17.5 19.5 20.5 23.5 21.5 26.0 28.5 xxxxx 24.0
15 Texel 1.06 x64 7.5 11.0 12.0 18.0 17.5 19.0 19.5 21.0 21.5 26.0 27.5 20.5 28.0 26.0 xxxxx
I have downloaded the full 600 games by your provided downloadlinkMikeB wrote:beram wrote:Thx for testing and downloadMikeB wrote:ok, this is my last update for the night:
you can check yourself for the latest updates here - about every 25 minutes or so: https://www.dropbox.com/s/q30k3kvul6qsa ... a.txt?dl=1Code: Select all
84 of 600 games completed... time control: 18000+300 Date: 10/30/16 : 03:12:46 84 game(s) loaded Rank Name Rating Δ + - # Σ Σ% W L D W% =% OppR --------------------------------------------------------------------------------------------------------- 1 Stockfish 102916-14y 3146 0.0 47 47 42 25.0 59.5 12 4 26 28.6 61.9 3085 2 Komodo 10.2 64-bit 3089 56.0 49 49 41 19.5 47.6 9 11 21 22.0 51.2 3104 3 Komodo 10.1 64-bit 3088 1.5 47 47 43 20.5 47.7 7 9 27 16.3 62.8 3103 4 Stockfish 281016 64 POPC 3077 11.0 49 49 42 19.0 45.2 8 12 22 19.0 52.4 3108 --------------------------------------------------------------------------------------------------------- 84 game(s) loaded
pgn file: https://www.dropbox.com/s/njswzxuo2u88gp2/all.pgn?dl=1
After 120 games, half point difference between 10.1 and 10.2
Code: Select all
Computer chess game 1 2 3 4 1 Stockfish 281016 64 POPC 21.0 - 19.021.5 - 18.524.0 - 16.0** 66.5/120 2 Stockfish 102916-14y 19.0 - 21.024.0 - 16.022.0 - 18.0 ** 65.0/120 3 Komodo 10.2 64-bit 18.5 - 21.516.0 - 24.020.0 - 20.0 ** 54.5/120 4 Komodo 10.1 64-bit 16.0 - 24.018.0 - 22.020.0 - 20.0 ** 54.0/120
after 444 games @ 3 min/3 sec/, the K10.2-K11.1 spread has widen to 17.3 ELO - certainly within the expected range for a limited number of game - will be stopping this run as it appears my last mod did not have a desired affect on SF ...also I misread Larry's comment about contempt and had it set to 15 for this run and I believe it may have impacted K's result against SF
Code: Select all
444 of 600 games completed... time control: 18000+300 Date: 10/30/16 : 15:39:37 444 game(s) loaded Rank Name Rating Δ + - # Σ Σ% W L D W% =% OppR --------------------------------------------------------------------------------------------------------- 1 Stockfish 281016 64 POPC 3145 0.0 20 20 223 131.5 59.0 64 24 135 28.7 60.5 3085 2 Stockfish 102916-14y 3129 16.3 20 20 222 124.0 55.9 56 30 136 25.2 61.3 3090 3 Komodo 10.2 64-bit 3072 57.3 20 20 221 98.0 44.3 34 59 128 15.4 57.9 3110 4 Komodo 10.1 64-bit 3054 17.3 20 20 222 90.5 40.8 29 70 123 13.1 55.4 3116 --------------------------------------------------------------------------------------------------------- 444 game(s) loaded
Code: Select all
Individual statistics:
1 Stockfish 301016 64 POPC : 3342 300 (+ 67,=214,- 19), 58.0 %
Komodo 10.2 64-bit : 100 (+ 33,= 64,- 3), 65.0 %
Komodo 10.1 64-bit : 100 (+ 29,= 64,- 7), 61.0 %
Stockfish 103016-14y : 100 (+ 5,= 86,- 9), 48.0 %
2 Stockfish 103016-14y : 3335 300 (+ 60,=220,- 20), 56.7 %
Komodo 10.2 64-bit : 100 (+ 25,= 66,- 9), 58.0 %
Komodo 10.1 64-bit : 100 (+ 26,= 68,- 6), 60.0 %
Stockfish 301016 64 POPC : 100 (+ 9,= 86,- 5), 52.0 %
3 Komodo 10.1 64-bit : 3264 300 (+ 25,=209,- 66), 43.2 %
Komodo 10.2 64-bit : 100 (+ 12,= 77,- 11), 50.5 %
Stockfish 103016-14y : 100 (+ 6,= 68,- 26), 40.0 %
Stockfish 301016 64 POPC : 100 (+ 7,= 64,- 29), 39.0 %
4 Komodo 10.2 64-bit : 3259 300 (+ 23,=207,- 70), 42.2 %
Komodo 10.1 64-bit : 100 (+ 11,= 77,- 12), 49.5 %
Stockfish 103016-14y : 100 (+ 9,= 66,- 25), 42.0 %
Stockfish 301016 64 POPC : 100 (+ 3,= 64,- 33), 35.0 %You are using as evident a run with a May 2016 version of Komodo (Komodo 10) versus a September 2016 version of Stockfish?beram wrote:Well perhaps you did relatively OK in the past but nowadays... this was latest result by Frank. A clear win for SF 180916 against all, Komodo 10 and 10.1 includedlkaufman wrote:I think we'll do ok relative to stockfish in tests like Frank's where each opponent plays against many other opponents, but we will not look so good in direct matches with Stockfish. The reasons for this are not all clear, although "Contempt" is a factor. It's up to each person to decide whether a direct match or a rr tournament is a better way to decide which of two engines is stronger. There are arguments both ways.S.Taylor wrote:What _I_ want to see is how this new komodo 10.2 does vs the SF that played in stage 3 of TCEC, OR, the version of SF that played in Graham Banks matches between sf and komodo.lkaufman wrote:We have released komodo 10.2 at komodochess.com. We estimate based on our testing that it is about 22 elo stronger than komodo 10.1 and about 30 elo stronger than komodo 10 at three minutes plus one second increment, on one or many threads. This is the largest elo gain we have had between versions since going to the subscription model. There were so many changes that it is very difficult to say which ones should get most of the credit. Evaluation, search, and time management were all changed significantly. Features (UCI options) are the same as in komodo 10.1. As usual, it is free to subscribers, and available at a 20% discount to those who bought komodo 9 or a later version.
It is very frustrating to have two moving targets at the same time.
e.g. let's say komodo beats ASM, how will we know if the older SF version, which Graham used, wouldn't have crushed komodo 10.2 like it did to komodo 10.1?
We should first test a moving target with a standing target before we go on to the second moving target.
(and besides Komodo 10.1 was performing 11 ELO points below K10 in his Test)
http://www.amateurschach.de/main/cross-tab/v438.htm
Code: Select all
1 SF 18Sep2016 BMI2 x64 C10 xxxxx 33.0 39.0 38.5 36.0 40.0 37.0 36.0 39.5 42.0 39.0 42.5 41.5 42.5 42.5 2 Komodo 10 x64 17.0 xxxxx 32.0 37.0 40.0 36.5 37.0 37.0 40.5 41.0 40.5 36.5 42.0 39.5 39.0 3 Houdini 4 STD B x64 11.0 18.0 xxxxx 26.5 26.0 27.5 32.5 29.5 30.5 34.0 32.0 33.0 28.5 32.0 38.0 4 Fire 4 x64 11.5 13.0 23.5 xxxxx 23.5 25.0 25.0 26.5 32.0 28.5 31.0 29.0 32.5 31.5 32.0 5 GullChess 3.0 BMI2 x64 14.0 10.0 24.0 26.5 xxxxx 31.0 23.5 24.5 28.0 30.5 30.0 32.0 26.5 31.5 32.5 6 Andscacs 0.872 BMI2 x64 10.0 13.5 22.5 25.0 19.0 xxxxx 26.0 25.0 26.0 23.5 26.5 31.5 25.0 28.5 31.0 7 Equinox 3.30 x64 13.0 13.0 17.5 25.0 26.5 24.0 xxxxx 24.0 23.5 26.0 29.0 27.5 28.5 32.5 30.5 8 Fizbo 1.8 BMI2 x64 14.0 13.0 20.5 23.5 25.5 25.0 26.0 xxxxx 29.5 23.5 25.5 27.0 28.0 30.5 29.0 9 Critter 1.6a x64 10.5 9.5 19.5 18.0 22.0 24.0 26.5 20.5 xxxxx 24.0 27.5 26.5 28.0 29.5 28.5 10 Fritz 15 x64 8.0 9.0 16.0 21.5 19.5 26.5 24.0 26.5 26.0 xxxxx 26.5 22.5 25.0 26.5 24.0 11 Nirvanachess 2.3 POP x64 11.0 9.5 18.0 19.0 20.0 23.5 21.0 24.5 22.5 23.5 xxxxx 26.5 27.0 28.5 22.5 12 Hannibal 1.7 x64 7.5 13.5 17.0 21.0 18.0 18.5 22.5 23.0 23.5 27.5 23.5 xxxxx 24.0 24.0 29.5 13 Chiron 3 x64 8.5 8.0 21.5 17.5 23.5 25.0 21.5 22.0 22.0 25.0 23.0 26.0 xxxxx 21.5 22.0 14 Protector 1.9.0 x64 7.5 10.5 18.0 18.5 18.5 21.5 17.5 19.5 20.5 23.5 21.5 26.0 28.5 xxxxx 24.0 15 Texel 1.06 x64 7.5 11.0 12.0 18.0 17.5 19.0 19.5 21.0 21.5 26.0 27.5 20.5 28.0 26.0 xxxxx
I am using as evidence that in Franks ultimate testing Komodo 10.1 mid july version is -10 ELO to Komodo 10 and the result in the matches against SF where about equal, SF won these with over 60%mjlef wrote:You are using as evident a run with a May 2016 version of Komodo (Komodo 10) versus a September 2016 version of Stockfish?beram wrote:Well perhaps you did relatively OK in the past but nowadays... this was latest result by Frank. A clear win for SF 180916 against all, Komodo 10 and 10.1 includedlkaufman wrote:I think we'll do ok relative to stockfish in tests like Frank's where each opponent plays against many other opponents, but we will not look so good in direct matches with Stockfish. The reasons for this are not all clear, although "Contempt" is a factor. It's up to each person to decide whether a direct match or a rr tournament is a better way to decide which of two engines is stronger. There are arguments both ways.S.Taylor wrote:What _I_ want to see is how this new komodo 10.2 does vs the SF that played in stage 3 of TCEC, OR, the version of SF that played in Graham Banks matches between sf and komodo.lkaufman wrote:We have released komodo 10.2 at komodochess.com. We estimate based on our testing that it is about 22 elo stronger than komodo 10.1 and about 30 elo stronger than komodo 10 at three minutes plus one second increment, on one or many threads. This is the largest elo gain we have had between versions since going to the subscription model. There were so many changes that it is very difficult to say which ones should get most of the credit. Evaluation, search, and time management were all changed significantly. Features (UCI options) are the same as in komodo 10.1. As usual, it is free to subscribers, and available at a 20% discount to those who bought komodo 9 or a later version.
It is very frustrating to have two moving targets at the same time.
e.g. let's say komodo beats ASM, how will we know if the older SF version, which Graham used, wouldn't have crushed komodo 10.2 like it did to komodo 10.1?
We should first test a moving target with a standing target before we go on to the second moving target.
(and besides Komodo 10.1 was performing 11 ELO points below K10 in his Test)
http://www.amateurschach.de/main/cross-tab/v438.htm
Code: Select all
1 SF 18Sep2016 BMI2 x64 C10 xxxxx 33.0 39.0 38.5 36.0 40.0 37.0 36.0 39.5 42.0 39.0 42.5 41.5 42.5 42.5 2 Komodo 10 x64 17.0 xxxxx 32.0 37.0 40.0 36.5 37.0 37.0 40.5 41.0 40.5 36.5 42.0 39.5 39.0 3 Houdini 4 STD B x64 11.0 18.0 xxxxx 26.5 26.0 27.5 32.5 29.5 30.5 34.0 32.0 33.0 28.5 32.0 38.0 4 Fire 4 x64 11.5 13.0 23.5 xxxxx 23.5 25.0 25.0 26.5 32.0 28.5 31.0 29.0 32.5 31.5 32.0 5 GullChess 3.0 BMI2 x64 14.0 10.0 24.0 26.5 xxxxx 31.0 23.5 24.5 28.0 30.5 30.0 32.0 26.5 31.5 32.5 6 Andscacs 0.872 BMI2 x64 10.0 13.5 22.5 25.0 19.0 xxxxx 26.0 25.0 26.0 23.5 26.5 31.5 25.0 28.5 31.0 7 Equinox 3.30 x64 13.0 13.0 17.5 25.0 26.5 24.0 xxxxx 24.0 23.5 26.0 29.0 27.5 28.5 32.5 30.5 8 Fizbo 1.8 BMI2 x64 14.0 13.0 20.5 23.5 25.5 25.0 26.0 xxxxx 29.5 23.5 25.5 27.0 28.0 30.5 29.0 9 Critter 1.6a x64 10.5 9.5 19.5 18.0 22.0 24.0 26.5 20.5 xxxxx 24.0 27.5 26.5 28.0 29.5 28.5 10 Fritz 15 x64 8.0 9.0 16.0 21.5 19.5 26.5 24.0 26.5 26.0 xxxxx 26.5 22.5 25.0 26.5 24.0 11 Nirvanachess 2.3 POP x64 11.0 9.5 18.0 19.0 20.0 23.5 21.0 24.5 22.5 23.5 xxxxx 26.5 27.0 28.5 22.5 12 Hannibal 1.7 x64 7.5 13.5 17.0 21.0 18.0 18.5 22.5 23.0 23.5 27.5 23.5 xxxxx 24.0 24.0 29.5 13 Chiron 3 x64 8.5 8.0 21.5 17.5 23.5 25.0 21.5 22.0 22.0 25.0 23.0 26.0 xxxxx 21.5 22.0 14 Protector 1.9.0 x64 7.5 10.5 18.0 18.5 18.5 21.5 17.5 19.5 20.5 23.5 21.5 26.0 28.5 xxxxx 24.0 15 Texel 1.06 x64 7.5 11.0 12.0 18.0 17.5 19.0 19.5 21.0 21.5 26.0 27.5 20.5 28.0 26.0 xxxxx
Also, as I understand, in the recent run with K 10.2, Contempt was improperly set to 15. 15 is a reasonable value for running against a larger number of mostly weaker opponents. A match against Stockfish or other Komodo versions should use a Contempt of 0.
The issue Mark is that every single tester keeps coming out with results that clearly show that Stockfish is stronger than Komodo. I personally know of a private match between the latest asmFish and Komodo 10.1 - and after about 700 games asmFish was up +90 Elo. I can confirm based on what the tester told me that it was a single thread, Nunn-match tournament at four minute\two second increment. Then you have reputable testers such as Ipman and SPCC clearly showing that Stockfish is stronger. Error margins and the amount of games played may be important; however, there are also things like common sense. The PATTERN of Stockfish's dominance has not changed for months. And that's exactly why Stockfish is now in the superfinal, and it's still undefeated in the TCEC tournament. Stockfish has gained massive strength this year. You should not just shrug off peoples' findings just because they may not be what you want to hear. The patterns speak for themselves.mjlef wrote:You are using as evident a run with a May 2016 version of Komodo (Komodo 10) versus a September 2016 version of Stockfish?beram wrote:Well perhaps you did relatively OK in the past but nowadays... this was latest result by Frank. A clear win for SF 180916 against all, Komodo 10 and 10.1 includedlkaufman wrote:I think we'll do ok relative to stockfish in tests like Frank's where each opponent plays against many other opponents, but we will not look so good in direct matches with Stockfish. The reasons for this are not all clear, although "Contempt" is a factor. It's up to each person to decide whether a direct match or a rr tournament is a better way to decide which of two engines is stronger. There are arguments both ways.S.Taylor wrote:What _I_ want to see is how this new komodo 10.2 does vs the SF that played in stage 3 of TCEC, OR, the version of SF that played in Graham Banks matches between sf and komodo.lkaufman wrote:We have released komodo 10.2 at komodochess.com. We estimate based on our testing that it is about 22 elo stronger than komodo 10.1 and about 30 elo stronger than komodo 10 at three minutes plus one second increment, on one or many threads. This is the largest elo gain we have had between versions since going to the subscription model. There were so many changes that it is very difficult to say which ones should get most of the credit. Evaluation, search, and time management were all changed significantly. Features (UCI options) are the same as in komodo 10.1. As usual, it is free to subscribers, and available at a 20% discount to those who bought komodo 9 or a later version.
It is very frustrating to have two moving targets at the same time.
e.g. let's say komodo beats ASM, how will we know if the older SF version, which Graham used, wouldn't have crushed komodo 10.2 like it did to komodo 10.1?
We should first test a moving target with a standing target before we go on to the second moving target.
(and besides Komodo 10.1 was performing 11 ELO points below K10 in his Test)
http://www.amateurschach.de/main/cross-tab/v438.htm
Code: Select all
1 SF 18Sep2016 BMI2 x64 C10 xxxxx 33.0 39.0 38.5 36.0 40.0 37.0 36.0 39.5 42.0 39.0 42.5 41.5 42.5 42.5 2 Komodo 10 x64 17.0 xxxxx 32.0 37.0 40.0 36.5 37.0 37.0 40.5 41.0 40.5 36.5 42.0 39.5 39.0 3 Houdini 4 STD B x64 11.0 18.0 xxxxx 26.5 26.0 27.5 32.5 29.5 30.5 34.0 32.0 33.0 28.5 32.0 38.0 4 Fire 4 x64 11.5 13.0 23.5 xxxxx 23.5 25.0 25.0 26.5 32.0 28.5 31.0 29.0 32.5 31.5 32.0 5 GullChess 3.0 BMI2 x64 14.0 10.0 24.0 26.5 xxxxx 31.0 23.5 24.5 28.0 30.5 30.0 32.0 26.5 31.5 32.5 6 Andscacs 0.872 BMI2 x64 10.0 13.5 22.5 25.0 19.0 xxxxx 26.0 25.0 26.0 23.5 26.5 31.5 25.0 28.5 31.0 7 Equinox 3.30 x64 13.0 13.0 17.5 25.0 26.5 24.0 xxxxx 24.0 23.5 26.0 29.0 27.5 28.5 32.5 30.5 8 Fizbo 1.8 BMI2 x64 14.0 13.0 20.5 23.5 25.5 25.0 26.0 xxxxx 29.5 23.5 25.5 27.0 28.0 30.5 29.0 9 Critter 1.6a x64 10.5 9.5 19.5 18.0 22.0 24.0 26.5 20.5 xxxxx 24.0 27.5 26.5 28.0 29.5 28.5 10 Fritz 15 x64 8.0 9.0 16.0 21.5 19.5 26.5 24.0 26.5 26.0 xxxxx 26.5 22.5 25.0 26.5 24.0 11 Nirvanachess 2.3 POP x64 11.0 9.5 18.0 19.0 20.0 23.5 21.0 24.5 22.5 23.5 xxxxx 26.5 27.0 28.5 22.5 12 Hannibal 1.7 x64 7.5 13.5 17.0 21.0 18.0 18.5 22.5 23.0 23.5 27.5 23.5 xxxxx 24.0 24.0 29.5 13 Chiron 3 x64 8.5 8.0 21.5 17.5 23.5 25.0 21.5 22.0 22.0 25.0 23.0 26.0 xxxxx 21.5 22.0 14 Protector 1.9.0 x64 7.5 10.5 18.0 18.5 18.5 21.5 17.5 19.5 20.5 23.5 21.5 26.0 28.5 xxxxx 24.0 15 Texel 1.06 x64 7.5 11.0 12.0 18.0 17.5 19.0 19.5 21.0 21.5 26.0 27.5 20.5 28.0 26.0 xxxxx
Also, as I understand, in the recent run with K 10.2, Contempt was improperly set to 15. 15 is a reasonable value for running against a larger number of mostly weaker opponents. A match against Stockfish or other Komodo versions should use a Contempt of 0.
The IPON list now reports Komodo 10.2 gained 22 elo over Komodo 10.1 (exactly matching our estimate in the readme and in our announcement here) with a 43 elo lead over Stockfish 7 at 5' + 3" with ponder on. I know that SF 8 will probably be something like 80 elo above SF 7 in bullet chess, but at IPON levels this may drop to somewhere near the 43 elo SF needs to top the list. In other words, it's too close to call whether Stockfish can regain the top spot if they release now. The Fast GM list has Komodo 10 2 at + 26 elo over Komodo 10.1 in bullet chess, and will probably show a lead of 50 elo or so over SF 7 on the ten minute list once it is available, so Stockfish 8 will have no easy time topping that list either. I readily admit that a new Stockfish will defeat Komodo 10.2 in direct matches (at least at fairly fast time controls), but this will not necessarily put Stockfish at the top of rating lists. Most of the testers seem to feel that performance against a gauntlet of dissimilar strong engines is more relevant than direct matches for judging strength, but of course this is a matter of opinion, not one of fact. Someday we'll figure out why Komodo underperforms against Stockfish, but it may just be something we can't change without hurting Komodo in overall performance. This is also the reason we do poorly on the IPMAN list, which has Komodo playing a zillion different Stockfish versions. It's useful for us to measure our own improvement this way, but it's not useful for predicting whether STockfish or Komodo will do better on lists like IPON, fastgm, Frank's list, CCRL, and CEGT. Doing well on lists like those is our primarily goal.APassionForCriminalJustic wrote:The issue Mark is that every single tester keeps coming out with results that clearly show that Stockfish is stronger than Komodo. I personally know of a private match between the latest asmFish and Komodo 10.1 - and after about 700 games asmFish was up +90 Elo. I can confirm based on what the tester told me that it was a single thread, Nunn-match tournament at four minute\two second increment. Then you have reputable testers such as Ipman and SPCC clearly showing that Stockfish is stronger. Error margins and the amount of games played may be important; however, there are also things like common sense. The PATTERN of Stockfish's dominance has not changed for months. And that's exactly why Stockfish is now in the superfinal, and it's still undefeated in the TCEC tournament. Stockfish has gained massive strength this year. You should not just shrug off peoples' findings just because they may not be what you want to hear. The patterns speak for themselves.mjlef wrote:You are using as evident a run with a May 2016 version of Komodo (Komodo 10) versus a September 2016 version of Stockfish?beram wrote:Well perhaps you did relatively OK in the past but nowadays... this was latest result by Frank. A clear win for SF 180916 against all, Komodo 10 and 10.1 includedlkaufman wrote:I think we'll do ok relative to stockfish in tests like Frank's where each opponent plays against many other opponents, but we will not look so good in direct matches with Stockfish. The reasons for this are not all clear, although "Contempt" is a factor. It's up to each person to decide whether a direct match or a rr tournament is a better way to decide which of two engines is stronger. There are arguments both ways.S.Taylor wrote:What _I_ want to see is how this new komodo 10.2 does vs the SF that played in stage 3 of TCEC, OR, the version of SF that played in Graham Banks matches between sf and komodo.lkaufman wrote:We have released komodo 10.2 at komodochess.com. We estimate based on our testing that it is about 22 elo stronger than komodo 10.1 and about 30 elo stronger than komodo 10 at three minutes plus one second increment, on one or many threads. This is the largest elo gain we have had between versions since going to the subscription model. There were so many changes that it is very difficult to say which ones should get most of the credit. Evaluation, search, and time management were all changed significantly. Features (UCI options) are the same as in komodo 10.1. As usual, it is free to subscribers, and available at a 20% discount to those who bought komodo 9 or a later version.
It is very frustrating to have two moving targets at the same time.
e.g. let's say komodo beats ASM, how will we know if the older SF version, which Graham used, wouldn't have crushed komodo 10.2 like it did to komodo 10.1?
We should first test a moving target with a standing target before we go on to the second moving target.
(and besides Komodo 10.1 was performing 11 ELO points below K10 in his Test)
http://www.amateurschach.de/main/cross-tab/v438.htm
Code: Select all
1 SF 18Sep2016 BMI2 x64 C10 xxxxx 33.0 39.0 38.5 36.0 40.0 37.0 36.0 39.5 42.0 39.0 42.5 41.5 42.5 42.5 2 Komodo 10 x64 17.0 xxxxx 32.0 37.0 40.0 36.5 37.0 37.0 40.5 41.0 40.5 36.5 42.0 39.5 39.0 3 Houdini 4 STD B x64 11.0 18.0 xxxxx 26.5 26.0 27.5 32.5 29.5 30.5 34.0 32.0 33.0 28.5 32.0 38.0 4 Fire 4 x64 11.5 13.0 23.5 xxxxx 23.5 25.0 25.0 26.5 32.0 28.5 31.0 29.0 32.5 31.5 32.0 5 GullChess 3.0 BMI2 x64 14.0 10.0 24.0 26.5 xxxxx 31.0 23.5 24.5 28.0 30.5 30.0 32.0 26.5 31.5 32.5 6 Andscacs 0.872 BMI2 x64 10.0 13.5 22.5 25.0 19.0 xxxxx 26.0 25.0 26.0 23.5 26.5 31.5 25.0 28.5 31.0 7 Equinox 3.30 x64 13.0 13.0 17.5 25.0 26.5 24.0 xxxxx 24.0 23.5 26.0 29.0 27.5 28.5 32.5 30.5 8 Fizbo 1.8 BMI2 x64 14.0 13.0 20.5 23.5 25.5 25.0 26.0 xxxxx 29.5 23.5 25.5 27.0 28.0 30.5 29.0 9 Critter 1.6a x64 10.5 9.5 19.5 18.0 22.0 24.0 26.5 20.5 xxxxx 24.0 27.5 26.5 28.0 29.5 28.5 10 Fritz 15 x64 8.0 9.0 16.0 21.5 19.5 26.5 24.0 26.5 26.0 xxxxx 26.5 22.5 25.0 26.5 24.0 11 Nirvanachess 2.3 POP x64 11.0 9.5 18.0 19.0 20.0 23.5 21.0 24.5 22.5 23.5 xxxxx 26.5 27.0 28.5 22.5 12 Hannibal 1.7 x64 7.5 13.5 17.0 21.0 18.0 18.5 22.5 23.0 23.5 27.5 23.5 xxxxx 24.0 24.0 29.5 13 Chiron 3 x64 8.5 8.0 21.5 17.5 23.5 25.0 21.5 22.0 22.0 25.0 23.0 26.0 xxxxx 21.5 22.0 14 Protector 1.9.0 x64 7.5 10.5 18.0 18.5 18.5 21.5 17.5 19.5 20.5 23.5 21.5 26.0 28.5 xxxxx 24.0 15 Texel 1.06 x64 7.5 11.0 12.0 18.0 17.5 19.0 19.5 21.0 21.5 26.0 27.5 20.5 28.0 26.0 xxxxx
Also, as I understand, in the recent run with K 10.2, Contempt was improperly set to 15. 15 is a reasonable value for running against a larger number of mostly weaker opponents. A match against Stockfish or other Komodo versions should use a Contempt of 0.
Look at some of what Ipman wrote concerning his Komodo development versions tested:
28-10-2016
"Getting here Komodo 1730.00 and after first 100games he has it difficult..need for sure a good result in next 100games..
Stockfish 221016 has in mean time 300games and still clear first engine!"
23-10-2016
"Komodo 1714.17 did it not so well and came at same level as K1702 (is removed) and is equal with K10.1
Komodo 1687 is still best version but has played with Dynamism=117
It's amazing but Komodo is already 50Elo behind best Stockfish! a lot improvement is needed!"
Then most recently: 30-10-2016
"Testing Komodo 1730.00 is stopped because not so good results and Komodo 10.2 is released!"
I love Komodo man. In effect, I hope that both you and Larry keep improving it. I love its positional style - and its excellent endgame play. That said, I think even Larry himself sees that Stockfish is on top now. But really, who cares? Just keep adding Elo. That is all that should matter. Stockfish has slowed down over the last few days; it is not impossible to catch Stockfish development over time.
I'm not trying to be provocative, but why isn't it as simple as this:lkaufman wrote:
I readily admit that a new Stockfish will defeat Komodo 10.2 in direct matches (at least at fairly fast time controls), but this will not necessarily put Stockfish at the top of rating lists. Most of the testers seem to feel that performance against a gauntlet of dissimilar strong engines is more relevant than direct matches for judging strength, but of course this is a matter of opinion, not one of fact. Someday we'll figure out why Komodo underperforms against Stockfish.