lkaufman wrote:CCRL 40/40 list now shows Komodo 10.1 tied for lead on 4 cpu with SF8, and Komodo 10.2 only 3 elo behind SF8 on one cpu. But we are not competitive in direct matches. I think this shows beyond much doubt that Komodo underperforms playing directly against SF compared to how they each do against a range of opponents. I expect Komodo 10.3 will top the 40/40 lists when it comes out, since we're virtually tied now.
Komodo is not tied on every list and is behind in http://fastgm.de/ for exemple.
Also Komodo seems to loose against Houdini 5, would you says it's underperform against it ?
I ask with no offense here, just want to know you opinion about it. I truly respect your work and your engine and I m happy you re still on top with your very small team
Both Stockfish 8 and Houdini 5 are stronger than Komodo 10.2 at blitz or near-blitz speeds. But at 40/40' it remains to be seen how Houdini 5 will fare; my guess is that it will not do better than Stockfish 8 did. Until we have significant data at that level it's too early to say much about Houdini 5.
Time for an update and reconsideration of what Larry wrote and stated about 20 weeks ago after the release of Komodo 10.2
[quote="lkaufman"]
The IPON list now reports Komodo 10.2 gained 22 elo over Komodo 10.1 (exactly matching our estimate in the readme and in our announcement here) with a 43 elo lead over Stockfish 7 at 5' + 3" with ponder on. I know that SF 8 will probably be something like 80 elo above SF 7 in bullet chess, but at IPON levels this may drop to somewhere near the 43 elo SF needs to top the list. It didnt happen Larry: SF8 gained +68ELO over SF7 on the Ipon and gap between Nr1 and nr3 K10.2 = 38ELO
In other words, it's too close to call whether Stockfish can regain the top spot if they release now. The Fast GM list has Komodo 10 2 at + 26 elo over Komodo 10.1 in bullet chess, and will probably show a lead of 50 elo or so over SF 7 on the ten minute list once it is available, so Stockfish 8 will have no easy time topping that list either. Yeah 26ELO at bullet chess, but only +3(!) over 10.1 at 10 minutes TC and 10.3 not much better (+8) over K10.2 and the 'easy time for SF8' was also there for SF8 +32 over K10.2http://www.fastgm.de/10min.html
I readily admit that a new Stockfish will defeat Komodo 10.2 in direct matches (at least at fairly fast time controls), but this will not necessarily put Stockfish at the top of rating lists. Sofar it did put on top SF8 everywhere, only H5 1core is on top at CCRL 40/40, and btw H5 is also still doing better on ratinglists than K10.2 and K10.3
Most of the testers seem to feel that performance against a gauntlet of dissimilar strong engines is more relevant than direct matches for judging strength, but of course this is a matter of opinion, not one of fact. Someday we'll figure out why Komodo underperforms against Stockfish, but it may just be something we can't change without hurting Komodo in overall performance. This is also the reason we do poorly on the IPMAN list, which has Komodo playing a zillion different Stockfish versions. There are also a zillion K versions playing and you are not doing poorly there
It's useful for us to measure our own improvement this way, but it's not useful for predicting whether STockfish or Komodo will do better on lists like IPON, fastgm, Frank's list, CCRL, and CEGT. Doing well on lists like those is our primarily goal. On all these lists IPON fastgm, CCRL, CEGT, SF8 is ahead of K10.2 (and K10.3) also 60% winning at LTC for SF8 over K10.2 (50 games at CEGT 40/120 +13 -1 =36 for SF8)
Also Houdini 5 wins with clear margin at LTC CEGT 40/120 +13 -3 =34 for H5, so the so called 'underperforming' is not only against SF8 but also against H5 at all time controls with 1 or with more cores[/quote]
I am a bit perplexed why this message was posted con concerning statements made many months ago. Stockfish 8 simply had a larger gain than what Larry expected. As I recall they were very active just before the Stockfish 8 release and the changes during this period helped them more than normal. Chess engine advances often come in clumps with several changes in a row being big gainers, followed by weeks of only tiny advances.
We can only estimate what these gains will be. Predicting the future will always be this way. You go with the best evidence you have, and sometimes you will be off. But I think Larry's history has overall been pretty amazing. We enjoy the competition, and it encourages us to work harder.
mjlef wrote:I am a bit perplexed why this message was posted con concerning statements made many months ago. Stockfish 8 simply had a larger gain than what Larry expected. As I recall they were very active just before the Stockfish 8 release and the changes during this period helped them more than normal. Chess engine advances often come in clumps with several changes in a row being big gainers, followed by weeks of only tiny advances.
We can only estimate what these gains will be. Predicting the future will always be this way. You go with the best evidence you have, and sometimes you will be off. But I think Larry's history has overall been pretty amazing. We enjoy the competition, and it encourages us to work harder.
Mark
No hard feelings Mark, good luck and succes with your hard labour you are doing both a good job. I certainly will buy a future version of Komodo