CCRL 40/40, 40/4 and FRC lists updated (28th February 2015)

[JJJ]

So Komodo 8 is 8 elo behind Stockfish 6 on 4 cores and 5 elo behind on 1 core, on the "40/40" list, which I think is really close to a 40/20 list on a modern I7, correct? Can Komodo 9 overcome this enormous deficit?

Hehe. On this list at this time controle, sure you can. Will Komodo 9 win in direct encounter versus stockfish 6 also ?

At bullet speeds SF6 still beats us by about 20 elo in latest tests, I would estimate that we would cross over 50% vs. SF6 at somewhere around the CCRL 40/40 (really 40/20) level, but I haven't confirmed that yet.

I doubt that Komodo9dev is allready able to beat SF6 at whatever TC
The performance of SF6 against K8 at LTC is average 57 % in 568 LTC games (lCCRL 40/40; CEGT; Tom Casanovas and Immortal chess)
57 % is about +50 ELO and since there has only been spoken of an improvement of +30 ELO for K9dev
Than that will not be enough to beat SF6

Code: Select all

SF6 vs K8 at LTC 	          games                    perc.  drawperc.  w/l ratio
TC 60m15s AMD FX8320 @8cpu	  188 games, +46 =120 -22	56,38%	63,83%	2,30
TC 40/40 CCRL @4cpu	            80 games +15 =60 −5	56,30%	75,00%	3,00
TC 30m5s Tom Casanovas @6cpu 	150 games  +40 =97 -13	59,00%	65,00%	3,08
TC 40/20 CEGT @4cpu	        100games +24 =67 -9	  57,50%	67,00%	2,67
TC 40/120 CEGT@1cpu	         50 games +12 =30 -8	  54,00%	60,00%	1,50
	                568 games +137 =374 -57 	avg 57,0%	avg 65,8%	avg 2,4

He didn't say how Komodo dev does actually against Stockfish. He said -20 in bullet only. He didn't say Komodo is actually 30 elo stronger. But I also saw him said he wanted at least 40 ELO in march. But I agree with you, I m afraid than 40 ELO is not enough to beat him in direct encounter.

[melajara]

Stephen Pohl made clear that actually he EXPECTED some regression with the first dev versions after the official SF6 as those dev versions are not benefiting from the same optimized compile as milestone 6, besides his list is computed from a relatively lame laptop not emphasizing any progress in MP scalability. Anyway, thank you for your answer. Strange that Martin is not saying anything about season 8.

[lkaufman]

Stephen Pohl made clear that actually he EXPECTED some regression with the first dev versions after the official SF6 as those dev versions are not benefiting from the same optimized compile as milestone 6, besides his list is computed from a relatively lame laptop not emphasizing any progress in MP scalability. Anyway, thank you for your answer. Strange that Martin is not saying anything about season 8.

Whether Stephen expected the regression doesn't matter; the bottom line is that someone who downloads the SF dev. version (up to the date he tested) will get a weaker engine than the official release (according to his results). As for MP, it is my understanding that the changes don't help with 4 cores or less, with 4 being the standard for most testing. Yes, with many cores they will presumably at least partially offset the advantage Komodo now has for such hardware.

[Jouni]

BTW in SF forum we see, that even Komodo 8 is beating Stockfish dev version in blitz:

Game in 4m+1s, hash 1028 8 move book repeating Win 7

Xeon

SF260218-Komodo8 16 core +1 elo 632 games
SF260218-Komodo8 20 core -14 elo 564 games
SF260218-Komodo8 24 core -28 elo 570 games

[lkaufman]

I guess this means that the MP patch in SF is not as effective on 16 cores as early reports suggested. I guess Komodo would be a big favorite if a new TCEC is held soon.

[APassionForCriminalJustic]

I guess this means that the MP patch in SF is not as effective on 16 cores as early reports suggested. I guess Komodo would be a big favorite if a new TCEC is held soon.

Whatever both you, Mark and Don (R.I.P.) have done with how you guys implemented SMP into Komodo is brilliant. Those tests stated above are showing that, indeed, Joona's patch is not as impactful as we had once thought.

[APassionForCriminalJustic]

So Komodo 8 is 8 elo behind Stockfish 6 on 4 cores and 5 elo behind on 1 core, on the "40/40" list, which I think is really close to a 40/20 list on a modern I7, correct? Can Komodo 9 overcome this enormous deficit?

Hehe. On this list at this time controle, sure you can. Will Komodo 9 win in direct encounter versus stockfish 6 also ?

At bullet speeds SF6 still beats us by about 20 elo in latest tests, I would estimate that we would cross over 50% vs. SF6 at somewhere around the CCRL 40/40 (really 40/20) level, but I haven't confirmed that yet.

I doubt that Komodo9dev is allready able to beat SF6 at whatever TC
The performance of SF6 against K8 at LTC is average 57 % in 568 LTC games (lCCRL 40/40; CEGT; Tom Casanovas and Immortal chess)
57 % is about +50 ELO and since there has only been spoken of an improvement of +30 ELO for K9dev
Than that will not be enough to beat SF6

Code: Select all

SF6 vs K8 at LTC 	          games                    perc.  drawperc.  w/l ratio
TC 60m15s AMD FX8320 @8cpu	  188 games, +46 =120 -22	56,38%	63,83%	2,30
TC 40/40 CCRL @4cpu	            80 games +15 =60 −5	56,30%	75,00%	3,00
TC 30m5s Tom Casanovas @6cpu 	150 games  +40 =97 -13	59,00%	65,00%	3,08
TC 40/20 CEGT @4cpu	        100games +24 =67 -9	  57,50%	67,00%	2,67
Tage single CEGT@1cpu	         50 games +12 =30 -8	  54,00%	60,00%	1,50
	                568 games +137 =374 -57 	avg 57,0%	avg 65,8%	avg 2,4

Your data is actually really nice; having an accumulation of all of these reputable rating lists is intelligent. Nonetheless, let us think critically here for one second. One of the certainties that the Komodo team believes is that Komodo's weakest time control versus Stockfish is bullet (60 second games). Larry has stated here that according to their testing Komodo development is only being beaten by 20 Elo points at bullet. When I had done about 60,000+ games for them Komodo was about 37 Elo weaker (bullet, and single-core games). If the advantage is only about 20 Elo for Stockfish 6 at Komodo's weakest time control then surpassing Stockfish 6 at longer LTC signatures will not be that difficult for Komodo 9. Evidently, Larry stating, "At bullet speeds SF6 still beats us by about 20 elo in latest tests" is very important information that does help contradict your argument because we know for a fact that Komodo scales much better at LTC time signatures and with more cores. Hence, Komodo development can be nowhere near 20 Elo weaker than Stockfish 6 at your above-stated time controls. Rest assured, the opposite will occur, Komodo will surpass Stockfish 6 at all reasonably long time controls, even if Komodo 9 was released today.

To further investigate my point even further, let us take a look at the CEGT TC 40/120 @1CPU. Now my data shows that 12 wins, eight losses and 30 draws equates to an Elo difference of +28 Elo. Remember that this is done with both engines running with only one core which should benefit Stockfish a lot. I think that +28 here is not a safe advantage at all for Stockfish, so expect Komodo 9 to come out on top due to the tournament time control listed. Moreover, as the cores increase, Komodo 9 will further increase its Elo advantage.

I believe that Komodo has proven its abilities at LTC; it has further proven that its SMP implementation(s) is superior for high-core count PC systems. I would easily expect that Komodo 9 will be on top once again, at least at reasonably long time controls. With greater than 16+ cores, Komodo 8 is still the better choice, and it is still stronger than Stockfish 6.

[beram]

So Komodo 8 is 8 elo behind Stockfish 6 on 4 cores and 5 elo behind on 1 core, on the "40/40" list, which I think is really close to a 40/20 list on a modern I7, correct? Can Komodo 9 overcome this enormous deficit?

Hehe. On this list at this time controle, sure you can. Will Komodo 9 win in direct encounter versus stockfish 6 also ?

At bullet speeds SF6 still beats us by about 20 elo in latest tests, I would estimate that we would cross over 50% vs. SF6 at somewhere around the CCRL 40/40 (really 40/20) level, but I haven't confirmed that yet.

I doubt that Komodo9dev is allready able to beat SF6 at whatever TC
The performance of SF6 against K8 at LTC is average 57 % in 568 LTC games (lCCRL 40/40; CEGT; Tom Casanovas and Immortal chess)
57 % is about +50 ELO and since there has only been spoken of an improvement of +30 ELO for K9dev
Than that will not be enough to beat SF6

Code: Select all

SF6 vs K8 at LTC 	          games                    perc.  drawperc.  w/l ratio
TC 60m15s AMD FX8320 @8cpu	  188 games, +46 =120 -22	56,38%	63,83%	2,30
TC 40/40 CCRL @4cpu	            80 games +15 =60 −5	56,30%	75,00%	3,00
TC 30m5s Tom Casanovas @6cpu 	150 games  +40 =97 -13	59,00%	65,00%	3,08
TC 40/20 CEGT @4cpu	        100games +24 =67 -9	  57,50%	67,00%	2,67
Tage single CEGT@1cpu	         50 games +12 =30 -8	  54,00%	60,00%	1,50
	                568 games +137 =374 -57 	avg 57,0%	avg 65,8%	avg 2,4

Your data is actually really nice; having an accumulation of all of these reputable rating lists is intelligent. Nonetheless, let us think critically here for one second. One of the certainties that the Komodo team believes is that Komodo's weakest time control versus Stockfish is bullet (60 second games). Larry has stated here that according to their testing Komodo development is only being beaten by 20 Elo points at bullet. When I had done about 60,000+ games for them Komodo was about 37 Elo weaker (bullet, and single-core games). If the advantage is only about 20 Elo for Stockfish 6 at Komodo's weakest time control then surpassing Stockfish 6 at longer LTC signatures will not be that difficult for Komodo 9. Evidently, Larry stating, "At bullet speeds SF6 still beats us by about 20 elo in latest tests" is very important information that does help contradict your argument because we know for a fact that Komodo scales much better at LTC time signatures and with more cores. Hence, Komodo development can be nowhere near 20 Elo weaker than Stockfish 6 at your above-stated time controls. Rest assured, the opposite will occur, Komodo will surpass Stockfish 6 at all reasonably long time controls, even if Komodo 9 was released today.

To further investigate my point even further, let us take a look at the CEGT TC 40/120 @1CPU. Now my data shows that 12 wins, eight losses and 30 draws equates to an Elo difference of +28 Elo. Remember that this is done with both engines running with only one core which should benefit Stockfish a lot. I think that +28 here is not a safe advantage at all for Stockfish, so expect Komodo 9 to come out on top due to the tournament time control listed. Moreover, as the cores increase, Komodo 9 will further increase its Elo advantage.

I believe that Komodo has proven its abilities at LTC; it has further proven that its SMP implementation(s) is superior for high-core count PC systems. I would easily expect that Komodo 9 will be on top once again, at least at reasonably long time controls. With greater than 16+ cores, Komodo 8 is still the better choice, and it is still stronger than Stockfish 6.

I disagree with you on the red marked sentences
1) there isnt that much of a lowering performance of SF 6 over K8 at longer time controls, outside the explanation range of higher draw percentage at LTC
2) when you just look at CEGT 40/20 and CCRL 40/40 you can observe that SF6 in contrary to what you state, SF6 performes not worse but better in matches against K8 when running on 4 cores instead of 1