Nakamura vs Stockfish, public match 8/23

[syzygy]

I agree with your comparison between SF5 and Fritz 8 on CCRL, but I think 375 Elo is too high for 5 speed doublings which includes going from 1 to 24 threads. 250 Elo might be more accurate.

And of course at least part of the Elo gained by multithreaded search should be attributed to software. It's not exactly trivial to make an engine run on many cores.

Hello Ronald )

Who said that since 2004 and 2014: the hardware speed difference should be 250 Elo ??

That was me. 5 doublings (as you say), each adding about 50 Elo instead of 75 Elo. 75 Elo seems too high even for a single-threaded doubling of nps. Going from 1 to 24 threads is expected to give a lot less Elo increase than a single-thread 24x speed increase.

Code: Select all

Engine                        Elo      Hardware Speed      Stockfish Bench
Deep Rybka 4.1 x64 6c        3358 Elo  i7 980X 4.33GHz       10512 kN/s 
Deep Rybka 4.1 x64 4c        3293 Elo  QX9650  3.66 GHz       4529 kN/s 
Rybka 4.1 x64 1c             3199 Elo  QX9650  3.66 GHz       1380 kN/s

Going from 4529 kN/s to 10512 kN/s is 65 Elo and about 1.215 doublings, which means 53.5 Elo per doubling. And here the number of threads only goes from 4 to 6.

Comparing the 2nd with the 3rd entry, I get 54.83 Elo per doubling.
Comparing the 1st with the 3rd entry, I get 54.28 Elo per doubling.

So maybe 54 Elo per doubling, which would mean 270 Elo for 5 doublings. But that is still too high if that includes going from 1 to 24 threads.

[Sedat Canbaz]

That was me. 5 doublings (as you say), each adding about 50 Elo instead of 75 Elo. 75 Elo seems too high even for a single-threaded doubling of nps. Going from 1 to 24 threads is expected to give a lot less Elo increase than a single-thread 24x speed increase.

So maybe 54 Elo per doubling, which would mean 270 Elo for 5 doublings. But that is still too high if that includes going from 1 to 24 threads.

Really ? )) nice...at least I am talking with expert )!

To be honest,
The doubling formula is wrong, I mean if we use for all engines exactly 50 Elo or 60 Elo or 70 Elo etc...
Because any chess engine Elo performance is hihgly depending on the used conditions...
Some chess engines perform even 35 Elo in case of doubling the processor speed
Even there are such engines where they gain more than 90 Elo in case of doubling

That's why I suggest to check at the game results....not at formulas )!

And here is another of my hardware speed tournaments:

Code: Select all

01 2xQX9775@4.0GHz      Rybka 3 x64 t8        3301   Elo      Majd Ansari
10 AMD 3400 2.40GHz     Rybka 3 w32 t1        3027   Elo      Sedat Canbaz

Note: The both hardwares Elo difference is approx. 280 Elo

And what about if we run Rybka 3 on 2x Xeon E5-2697 v2 @3.22 GHz ??
I expect to see at least 100 Elo in favor...and Rybka 3 probably will be rated at least 3400 Elo
So again the Elo difference (since 2004-2014) shows approx. 380 Elo

Hopes this time helps...

[Uri Blass]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

I think that playing against humans is different.
chess programs do not know that they play against stronger opponent and do not adopt their style.

Top humans know that they play against a stronger opponent and
I think that they can play for a draw and get it in part of the games
so I do not believe rating against humans is around 3500.

[Sedat Canbaz]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

I think that playing against humans is different.
chess programs do not know that they play against stronger opponent and do not adopt their style.

Top humans know that they play against a stronger opponent and
I think that they can play for a draw and get it in part of the games
so I do not believe rating against humans is around 3500.

Hmm...

You are missing again one very important point:
-Engine Top Book Makers know the weakness of GMs !

Why nobody is contacting me...???
You know... I already announced a challenge

Since now...I will increase from 10% to 20 %
So, I promise to give 20 % commision for that person who will find me a GM !


Let's see...what will be the Elo difference
I expect to see at least 700 Elo difference in favor for Stockfish+ superior book + decent fast hardware

[Laskos]

Anyway this discussion can go on for ever and we will never agree.

No, it's not a matter of which colour one prefers. It's a matter that you claimed that 600 Elo points gain from software is less than 250 Elo points gain from hardware during the past 10 years. There is 350 Elo points difference in our claims. That is, 5 doublings at 40/4', or a factor of 32, or 3200% speed-up. You claim that things like POPCOUNT, which give a speed-up of less than 10%, are making up for those 3200%. Your claim is simply wrong, and while everyone is free to have opinions, I would suggest to keep your opinions for yourself, and not dis-inform people on a forum.

So, as a conclusion, in the last 10 years software improvement is about 600 Elo points, hardware about 250 Elo points.

I said software improvement <= 350 Elo for the last ten years, I never spoke about < 250 Elo anywhere.

And I like to vent my opinions whenever I want, I don't need any suggestions from you in this respect.

You argued for 350 Elo points gain from software for 20 year span, which is totally unreasonable, unfounded, and misinforming.
I now show CEGT 40/4' ratings (CEGT uses Ordo, with its logistical Elos and no rating compression).

The best of 2014:
Stockfish 5.0 x64 1CPU 3158

The best of 2004:
Shredder 8 2517

640 Elo gain from software in 10 years, maybe one can reduce it to 600 arguing about software optimization for modern hardware (although many in rating groups like CEGT and CCRL use a moderately aged hardware).

[Uri Blass]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

I think that playing against humans is different.
chess programs do not know that they play against stronger opponent and do not adopt their style.

Top humans know that they play against a stronger opponent and
I think that they can play for a draw and get it in part of the games
so I do not believe rating against humans is around 3500.

Hmm...

You are missing again one very important point:
-Engine Top Book Makers know the weakness of GMs !

Why nobody is contacting me...???
You know... I already announced a challenge

Since now...I will increase from 10% to 20 %
So, I promise to give 20 % commision for that person who will find me a GM !


Let's see...what will be the Elo difference
I expect to see at least 700 Elo difference in favor for Stockfish+ superior book + decent fast hardware

http://www.fide.com/fide/handbook.html? ... ew=article

fide handbook has the following expected results(see 8.1a):
800 elo difference 100%
677 elo difference 99%
589 elo difference 98%
538 elo difference 97%
501 elo difference 96%
470 elo difference 95%

I think that it may be interesting if somebody suggest a challange for humans to get 5% against chess engines in a match of 100 games.

I guess that you expect GM's to be unable to get even 2% in 100 games based on rating of 3500.

[Dr.Wael Deeb]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

I think that playing against humans is different.
chess programs do not know that they play against stronger opponent and do not adopt their style.

Top humans know that they play against a stronger opponent and
I think that they can play for a draw and get it in part of the games
so I do not believe rating against humans is around 3500.

Hmm...

You are missing again one very important point:
-Engine Top Book Makers know the weakness of GMs !

Why nobody is contacting me...???
You know... I already announced a challenge

Since now...I will increase from 10% to 20 %
So, I promise to give 20 % commision for that person who will find me a GM !


Let's see...what will be the Elo difference
I expect to see at least 700 Elo difference in favor for Stockfish+ superior book + decent fast hardware

I'll go with around 500 difference my friend but all in all I agree with you....
Dr.D

[Dr.Wael Deeb]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

I think that playing against humans is different.
chess programs do not know that they play against stronger opponent and do not adopt their style.

Top humans know that they play against a stronger opponent and
I think that they can play for a draw and get it in part of the games
so I do not believe rating against humans is around 3500.

Hmm...

You are missing again one very important point:
-Engine Top Book Makers know the weakness of GMs !

Why nobody is contacting me...???
You know... I already announced a challenge

Since now...I will increase from 10% to 20 %
So, I promise to give 20 % commision for that person who will find me a GM !


Let's see...what will be the Elo difference
I expect to see at least 700 Elo difference in favor for Stockfish+ superior book + decent fast hardware

http://www.fide.com/fide/handbook.html? ... ew=article

fide handbook has the following expected results(see 8.1a):
800 elo difference 100%
677 elo difference 99%
589 elo difference 98%
538 elo difference 97%
501 elo difference 96%
470 elo difference 95%

I think that it may be interesting if somebody suggest a challange for humans to get 5% against chess engines in a match of 100 games.

I guess that you expect GM's to be unable to get even 2% in 100 games based on rating of 3500.

!00 games are too much for one human to play Uri....

Maybe you mean several top GMs will play a total of 100 games
Dr.D

[Sedat Canbaz]

I believe that there were some years when most of the improvement were due to hardware but it stopped in 2004 when fabien released fruit and if you look at the progress in software in the last 10 years then you will find that it is clearly bigger than the improvement in hardware in the last 10 years.

Indeed. Just take Fruit 2.1 which must have been released somewhere in 2005 and at that time was at least close to the top. On CCRL 40/40 it has a rating of 2693. SF5 single cpu has a rating of 3212. That is more than 500 Elo software improvement in about 9 years.

Yes... but as far as I know,
Uri disagrees that Stockfish 5 should be rated around 3500 Elo (on decent hardwares )))

His estimation is around 3200 Elo ))

I think that playing against humans is different.
chess programs do not know that they play against stronger opponent and do not adopt their style.

Top humans know that they play against a stronger opponent and
I think that they can play for a draw and get it in part of the games
so I do not believe rating against humans is around 3500.

Hmm...

You are missing again one very important point:
-Engine Top Book Makers know the weakness of GMs !

Why nobody is contacting me...???
You know... I already announced a challenge

Since now...I will increase from 10% to 20 %
So, I promise to give 20 % commision for that person who will find me a GM !


Let's see...what will be the Elo difference
I expect to see at least 700 Elo difference in favor for Stockfish+ superior book + decent fast hardware

http://www.fide.com/fide/handbook.html? ... ew=article

fide handbook has the following expected results(see 8.1a):
800 elo difference 100%
677 elo difference 99%
589 elo difference 98%
538 elo difference 97%
501 elo difference 96%
470 elo difference 95%

I think that it may be interesting if somebody suggest a challange for humans to get 5% against chess engines in a match of 100 games.

I guess that you expect GM's to be unable to get even 2% in 100 games based on rating of 3500.

Well...

To be honest,
I think that GMs will be unable to get even 5% in 100 games


For example, here is SCCT Open-Source Rating

Note: I picked only those Stockfish's opponents, which are estimated to be at near level as many GMs)

And as we see Stockfish's winning percentage is 96.3%

Code: Select all

   # PLAYER                  : RATING    POINTS  PLAYED    (%)
   1 Stockfish 210614 x64    : 3396.0     578.0     600   96.3%
   2 Crafty 24 x64           : 2900.0     350.5     600   58.4%
   3 Scorpio 2.7.7 x64       : 2886.5     341.0     600   56.8%
   4 Cheng4 0.36c x64        : 2871.1     328.0     600   54.7%
   5 Arasan 17.2 x64         : 2860.3     318.5     600   53.1%
   6 Octochess r5190 x64     : 2849.4     309.0     600   51.5%
   7 EXchess v7.26a          : 2843.0     305.0     600   50.8%
   8 Rodent 1.4 x64          : 2825.1     290.0     600   48.3%
   9 Bobcat 3.25 x64         : 2784.8     256.5     600   42.8%
  10 Bison 9.11 x64          : 2762.6     238.5     600   39.8%
  11 Daydreamer 1.75 x64     : 2709.2     196.5     600   32.8%
  12 Redqueen 1.1.97 x64     : 2708.6     196.0     600   32.7%
  13 Cyrano 0.6b17 x64       : 2704.0     192.5     600   32.1%

More Details:
-Stockfish is played as one core (between 1 core and 24 core there is approx. 200 Elo)
-Stockfish is played with short neutral book, but with well-optimized book we can expect at least +100 Elo (in favor for Stockfish)
-Stockfish 210614 has been used...the latest releases are at least 20 Elo stronger

And then just imagine...))

Ok... let's forget everything...
But be sure of this: Stockfish 24 core will be rated at least 600 Elo stronger than GMs (at slow time control)

In Blitz, Stockfish 24 core is expecting... at least 700 Elo stronger than GMs

[Sedat Canbaz]

Continuing...

Why GMs preferred to use Rybka 3 as a help against Stockfish 5 ?
Why GMs preferred to use Rybka 3 + book against Stockfish 5+ without book ?
Why GMs did not prefer to use Stockfish 5 on latest fast decent macines?

Simply because GMs perfectly know that in case of Stockfish + book + fast hardware etc...
Stockfish can be at at least 300 Elo stronger, in other words unbeatable

Yes...GMs tried to find the weakness in Stockfish...
And even after such disadvantages...Stockfish could not be destroyed in any single game !

Well- done to the creators of Stockfish !!