Komodo 10.2 released.

[beram]

CCRL 40/40 list now shows Komodo 10.1 tied for lead on 4 cpu with SF8, and Komodo 10.2 only 3 elo behind SF8 on one cpu. But we are not competitive in direct matches. I think this shows beyond much doubt that Komodo underperforms playing directly against SF compared to how they each do against a range of opponents. I expect Komodo 10.3 will top the 40/40 lists when it comes out, since we're virtually tied now.

Komodo is not tied on every list and is behind in http://fastgm.de/ for exemple.

Also Komodo seems to loose against Houdini 5, would you says it's underperform against it ?

I ask with no offense here, just want to know you opinion about it. I truly respect your work and your engine and I m happy you re still on top with your very small team

Both Stockfish 8 and Houdini 5 are stronger than Komodo 10.2 at blitz or near-blitz speeds. But at 40/40' it remains to be seen how Houdini 5 will fare; my guess is that it will not do better than Stockfish 8 did. Until we have significant data at that level it's too early to say much about Houdini 5.

Well atm it seems to be fairing as well or even better than SF8 when you look at the match by Graham with Tom live chess viewer http://www.talkchess.com/forum/viewtopic.php?t=62184
standing so far +7 -1 = 23

Code: Select all

 RANK   ENGINE                                    GAMES  POINTS   1                               2                               
----------------------------------------------------------------------------------------------------------------------------------
   1.   HOUDINI 5.01 64-BIT                         31    18.5    ******************************* =========1======11===11=1===1=0 
   2.   KOMODO 10.2 64-BIT                          31    12.5    =========0======00===00=0===0=1 ******************************* 

Total games = 31

[beram]

Time for an update and reconsideration of what Larry wrote and stated about 20 weeks ago after the release of Komodo 10.2
[quote="lkaufman"]

The IPON list now reports Komodo 10.2 gained 22 elo over Komodo 10.1 (exactly matching our estimate in the readme and in our announcement here) with a 43 elo lead over Stockfish 7 at 5' + 3" with ponder on. I know that SF 8 will probably be something like 80 elo above SF 7 in bullet chess, but at IPON levels this may drop to somewhere near the 43 elo SF needs to top the list.
It didnt happen Larry: SF8 gained +68ELO over SF7 on the Ipon and gap between Nr1 and nr3 K10.2 = 38ELO

Code: Select all

IPON - Complete list 2017.01.09:

   # PLAYER                       : RATING  ERROR     (%)    D(%)  OppAvg   CFS(next)    POINTS       W       D       L  PLAYED
   1 Stockfish 8                  :   3300     10   80.2%    36.3    3037      99        3530.5    2732    1597      71    4400
   2 Houdini 5.01                 :   3285     11   78.7%    35.9    3039     100        3118.0    2408    1420     132    3960
   3 Komodo 10.2                  :   3262      9   76.1%    36.0    3042      85        3682.5    2812    1741     287    4840
   4 Komodo 10.1                  :   3255     10   80.1%    30.9    2997      81        3522.5    2843    1359     198    4400
   5 Komodo 10                    :   3249     11   81.0%    30.1    2979      96        3562.5    2901    1323     176    4400
   6 Komodo 9.42                  :   3236     11   80.4%    30.9    2973      75        3362.0    2717    1290     173    4180
   7 Stockfish 7                  :   3232      8   76.5%    38.9    3004     100        5719.0    4264    2910     306    7480

In other words, it's too close to call whether Stockfish can regain the top spot if they release now. The Fast GM list has Komodo 10 2 at + 26 elo over Komodo 10.1 in bullet chess, and will probably show a lead of 50 elo or so over SF 7 on the ten minute list once it is available, so Stockfish 8 will have no easy time topping that list either.
Yeah 26ELO at bullet chess, but only +3(!) over 10.1 at 10 minutes TC and 10.3 not much better (+8) over K10.2 and the 'easy time for SF8' was also there for SF8 +32 over K10.2 http://www.fastgm.de/10min.html

Code: Select all

Complete List: 02.02.2017

     Program               :    Elo   Error  Played   (%)       W       D       L     D(%)   CFS(next)
 ------------------------------------------------------------------------------------------------------
   1 Stockfish 8           :   3239      7    6300   76.9%    3488    2718      94    43.1      99    
   2 Houdini 5.01          :   3227      8    6000   75.8%    3273    2556     171    42.6      98    
   3 Komodo 10.3           :   3215      8    4200   72.0%    2074    1898     228    45.2      95    
   4 Komodo 10.2           :   3207      7    5400   70.8%    2574    2500     326    46.3      70    
   5 Komodo 10.1           :   3204      9    4200   77.8%    2485    1569     146    37.4      73  

I readily admit that a new Stockfish will defeat Komodo 10.2 in direct matches (at least at fairly fast time controls), but this will not necessarily put Stockfish at the top of rating lists.
Sofar it did put on top SF8 everywhere, only H5 1core is on top at CCRL 40/40, and btw H5 is also still doing better on ratinglists than K10.2 and K10.3

Most of the testers seem to feel that performance against a gauntlet of dissimilar strong engines is more relevant than direct matches for judging strength, but of course this is a matter of opinion, not one of fact. Someday we'll figure out why Komodo underperforms against Stockfish, but it may just be something we can't change without hurting Komodo in overall performance. This is also the reason we do poorly on the IPMAN list, which has Komodo playing a zillion different Stockfish versions.
There are also a zillion K versions playing and you are not doing poorly there

It's useful for us to measure our own improvement this way, but it's not useful for predicting whether STockfish or Komodo will do better on lists like IPON, fastgm, Frank's list, CCRL, and CEGT. Doing well on lists like those is our primarily goal.
On all these lists IPON fastgm, CCRL, CEGT, SF8 is ahead of K10.2 (and K10.3) also 60% winning at LTC for SF8 over K10.2 (50 games at CEGT 40/120 +13 -1 =36 for SF8)
Also Houdini 5 wins with clear margin at LTC CEGT 40/120 +13 -3 =34 for H5, so the so called 'underperforming' is not only against SF8 but also against H5 at all time controls with 1 or with more cores[/quote]

[mjlef]

I am a bit perplexed why this message was posted con concerning statements made many months ago. Stockfish 8 simply had a larger gain than what Larry expected. As I recall they were very active just before the Stockfish 8 release and the changes during this period helped them more than normal. Chess engine advances often come in clumps with several changes in a row being big gainers, followed by weeks of only tiny advances.


We can only estimate what these gains will be. Predicting the future will always be this way. You go with the best evidence you have, and sometimes you will be off. But I think Larry's history has overall been pretty amazing. We enjoy the competition, and it encourages us to work harder.

Mark

[beram]

I am a bit perplexed why this message was posted con concerning statements made many months ago. Stockfish 8 simply had a larger gain than what Larry expected. As I recall they were very active just before the Stockfish 8 release and the changes during this period helped them more than normal. Chess engine advances often come in clumps with several changes in a row being big gainers, followed by weeks of only tiny advances.


We can only estimate what these gains will be. Predicting the future will always be this way. You go with the best evidence you have, and sometimes you will be off. But I think Larry's history has overall been pretty amazing. We enjoy the competition, and it encourages us to work harder.

Mark

No hard feelings Mark, good luck and succes with your hard labour you are doing both a good job. I certainly will buy a future version of Komodo

grts Bram

[JJJ]

Since Stockfish dev improved only by 10 elo in 3 month, the next Komodo has to win around 40/50 elo to catch it fully :

http://fastgm.de/60min.html

But of course at bullet, the gap is wider.