CCRL 40/4 lists updated (11th August 2012)

[Sedat Canbaz]

Our data for Komodo gave 90 per doubling, which is 1.3% per elo, which means 10% = 13 elo. But of course the other engines also gain something from SSE, maybe 5 elo, so we are back to 8 elo assuming both engines use SSE. But some of the opposing engines don't have SSE capability, so this means the value of SSE for Komodo is between 8 and 13, let's say 10. Yes, there is some decrease in the value of each doubling with greater depth for a given engine, but it seems that as the engines improve the value of doubling has not been shrinking. It all depends on the nature of the improvement. Since the doubling value has been calculated from recent versions where the range of speed is fairly small, I think this factor is negligible for the present calculation.

Hello dear Larry,

Of course i give you right,you and Don are the owners of Komodo...
That's why probably you know better than all about Komodo's Elo speed doubling

Btw,
I have no much info or idea about the rest people hardware speed Elo testings...
But however,i strongly believe that the right Hardware speed Elo measuring method should be done in Auto232 mode

For example,i have some Auto232 Hardware speed Elo testings:
http://www.sedatcanbaz.com/chess/ratings/scct-auto232/

Where i've noticed that any Engine has different Elo performance,in case of doubling the processor speed
Especially Naum 6 core has the highest Elo speed difference:90 Elo
The lowest Elo difference is Fire 2.2 xTreme x64 6c,just 36 Elo
It seems,some critical engines Elo performance suffer with many cores

Code: Select all

Rank  Name                     Elo    +    -  games  score oppo. draws
1 Houdini 2.0c Pro x64 6c      3423   19   18  1008   70%  3261   42% 
5 Houdini 2.0 Pro x64 4c       3363   16   16  1131   61%  3294   49% 
*Elo Difference:60


6 Deep Rybka 4.1 x64 6c         3358   15   15  1363   59%  3305   54%
15 Deep Rybka 4.1 x64 4c        3293   13   14  1602   47%  3314   56% 
*Elo Difference:65

12 Fire 2.2 xTreme x64 6c       3311   23   22   530   52%  3297   63% 
18 Fire 2.2 xTreme x64 4c       3275   16   16  1165   41%  3328   59%
*Elo Difference:36

22 Naum 4.2 x64 6c              3266   22   22   631   50%  3261   46% 
42 Naum 4.2 x64 4c              3176   17   18  1008   35%  3270   45% 
*Elo Difference:90

Houdini 2.0 BenchMarks:

Code: Select all

kN/s    Mate Processor              Speed  Cores L.P  EXE  Hardware User
21416   12s  Intel Core i7 980X     @4.33GHz  6  OFF  x64  Sedat Canbaz
10872   47s  Intel Core i7 920      @3.00GHz  4  OFF  x64  Sedat Canbaz
10336   49s  Intel Extreme QX9650   @3.66GHz  4  OFF  x64  Sedat Canbaz

6 Cores = MP Engines are played on my six core machine (i7 980X @4.33GHz)
4 Cores = MP Engines are played on my Quads (i7 920 @3.0GHz/QX9650 @3.66GHz)


Greetings,
Sedat

[Uri Blass]

Our data for Komodo gave 90 per doubling, which is 1.3% per elo, which means 10% = 13 elo. But of course the other engines also gain something from SSE, maybe 5 elo, so we are back to 8 elo assuming both engines use SSE. But some of the opposing engines don't have SSE capability, so this means the value of SSE for Komodo is between 8 and 13, let's say 10. Yes, there is some decrease in the value of each doubling with greater depth for a given engine, but it seems that as the engines improve the value of doubling has not been shrinking. It all depends on the nature of the improvement. Since the doubling value has been calculated from recent versions where the range of speed is fairly small, I think this factor is negligible for the present calculation.

Hello dear Larry,

Of course i give you right,you and Don are the owners of Komodo...
That's why probably you know better than all about Komodo's Elo speed doubling

Btw,
I have no much info or idea about the rest people hardware speed Elo testings...
But however,i strongly believe that the right Hardware speed Elo measuring method should be done in Auto232 mode

For example,i have some Auto232 Hardware speed Elo testings:
http://www.sedatcanbaz.com/chess/ratings/scct-auto232/

Where i've noticed that any Engine has different Elo performance,in case of doubling the processor speed
Especially Naum 6 core has the highest Elo speed difference:90 Elo
The lowest Elo difference is Fire 2.2 xTreme x64 6c,just 36 Elo
It seems,some critical engines Elo performance suffer with many cores

Code: Select all

Rank  Name                     Elo    +    -  games  score oppo. draws
1 Houdini 2.0c Pro x64 6c      3423   19   18  1008   70%  3261   42% 
5 Houdini 2.0 Pro x64 4c       3363   16   16  1131   61%  3294   49% 
*Elo Difference:60


6 Deep Rybka 4.1 x64 6c         3358   15   15  1363   59%  3305   54%
15 Deep Rybka 4.1 x64 4c        3293   13   14  1602   47%  3314   56% 
*Elo Difference:65

12 Fire 2.2 xTreme x64 6c       3311   23   22   530   52%  3297   63% 
18 Fire 2.2 xTreme x64 4c       3275   16   16  1165   41%  3328   59%
*Elo Difference:36

22 Naum 4.2 x64 6c              3266   22   22   631   50%  3261   46% 
42 Naum 4.2 x64 4c              3176   17   18  1008   35%  3270   45% 
*Elo Difference:90

Houdini 2.0 BenchMarks:

Code: Select all

kN/s    Mate Processor              Speed  Cores L.P  EXE  Hardware User
21416   12s  Intel Core i7 980X     @4.33GHz  6  OFF  x64  Sedat Canbaz
10872   47s  Intel Core i7 920      @3.00GHz  4  OFF  x64  Sedat Canbaz
10336   49s  Intel Extreme QX9650   @3.66GHz  4  OFF  x64  Sedat Canbaz

6 Cores = MP Engines are played only on my six core machine (i7 980X @4.33GHz)
4 Cores = MP Engines are played only on my Quads (i7 920 @3.0GHz/QX9650 @3.66GHz)


Greetings,
Sedat

When you compare 6 cores and 4 cores
then it is not equivalent to multiplying the processor speed by some number.

Some engines may earn more from more cores and some engines may earn less from more cores.

This is clearly not relevant for the komodo discussion because Komodo does not use more than one core.

[Sedat Canbaz]

Our data for Komodo gave 90 per doubling, which is 1.3% per elo, which means 10% = 13 elo. But of course the other engines also gain something from SSE, maybe 5 elo, so we are back to 8 elo assuming both engines use SSE. But some of the opposing engines don't have SSE capability, so this means the value of SSE for Komodo is between 8 and 13, let's say 10. Yes, there is some decrease in the value of each doubling with greater depth for a given engine, but it seems that as the engines improve the value of doubling has not been shrinking. It all depends on the nature of the improvement. Since the doubling value has been calculated from recent versions where the range of speed is fairly small, I think this factor is negligible for the present calculation.

Hello dear Larry,

Of course i give you right,you and Don are the owners of Komodo...
That's why probably you know better than all about Komodo's Elo speed doubling

Btw,
I have no much info or idea about the rest people hardware speed Elo testings...
But however,i strongly believe that the right Hardware speed Elo measuring method should be done in Auto232 mode

For example,i have some Auto232 Hardware speed Elo testings:
http://www.sedatcanbaz.com/chess/ratings/scct-auto232/

Where i've noticed that any Engine has different Elo performance,in case of doubling the processor speed
Especially Naum 6 core has the highest Elo speed difference:90 Elo
The lowest Elo difference is Fire 2.2 xTreme x64 6c,just 36 Elo
It seems,some critical engines Elo performance suffer with many cores

Code: Select all

Rank  Name                     Elo    +    -  games  score oppo. draws
1 Houdini 2.0c Pro x64 6c      3423   19   18  1008   70%  3261   42% 
5 Houdini 2.0 Pro x64 4c       3363   16   16  1131   61%  3294   49% 
*Elo Difference:60


6 Deep Rybka 4.1 x64 6c         3358   15   15  1363   59%  3305   54%
15 Deep Rybka 4.1 x64 4c        3293   13   14  1602   47%  3314   56% 
*Elo Difference:65

12 Fire 2.2 xTreme x64 6c       3311   23   22   530   52%  3297   63% 
18 Fire 2.2 xTreme x64 4c       3275   16   16  1165   41%  3328   59%
*Elo Difference:36

22 Naum 4.2 x64 6c              3266   22   22   631   50%  3261   46% 
42 Naum 4.2 x64 4c              3176   17   18  1008   35%  3270   45% 
*Elo Difference:90

Houdini 2.0 BenchMarks:

Code: Select all

kN/s    Mate Processor              Speed  Cores L.P  EXE  Hardware User
21416   12s  Intel Core i7 980X     @4.33GHz  6  OFF  x64  Sedat Canbaz
10872   47s  Intel Core i7 920      @3.00GHz  4  OFF  x64  Sedat Canbaz
10336   49s  Intel Extreme QX9650   @3.66GHz  4  OFF  x64  Sedat Canbaz

6 Cores = MP Engines are played only on my six core machine (i7 980X @4.33GHz)
4 Cores = MP Engines are played only on my Quads (i7 920 @3.0GHz/QX9650 @3.66GHz)


Greetings,
Sedat

When you compare 6 cores and 4 cores
then it is not equivalent to multiplying the processor speed by some number.

Some engines may earn more from more cores and some engines may earn less from more cores.

This is clearly not relevant for the komodo discussion because Komodo does not use more than one core.

Probably you missed to read my previous statement:
It seems,some critical engines Elo performance suffer with many cores

In my opinion,
My hardware Elo testing is not bad,even i can say it contains a very useful and valuable information

And i think there is no much interest or pointless to measure a slow Quad with faster Quad
Because nowadays,almost all experienced chess fans make no any updates e.g from Q6600 to i7 2600k

Or is there another six core machine,which is x2 times faster than i7 980X @4.33GHz ?

In other words,
If you check the latest 10-15 years,you will noticed that in every 2-3 years we see new processors,which are capable to use many cores

Btw,now we are in 2012,
And i think its time to concentrate on Elo Hardware speed differences based latest MP engines+fastest machines


Hope this helps,
Sedat

[Modern Times]

The only way to do this is to run an off-line calculation on the entire CCRL database, with Komodo 5 separated out between AMD and Intel. But with just 500 games roughly for each, the statistical error margins will be huge. If I get time I will post the result here, but in any case there is no substitute for some proper, controlled testing on this issue.

I have now done this. The CCRL database is of course connected, any non-connected engines are automatically excluded.

Intel Non-SSE shows as +15 Elo over AMD SSE4. But with error margins +/- 25 Elo it is inconclusive.

[Sedat Canbaz]

The only way to do this is to run an off-line calculation on the entire CCRL database, with Komodo 5 separated out between AMD and Intel. But with just 500 games roughly for each, the statistical error margins will be huge. If I get time I will post the result here, but in any case there is no substitute for some proper, controlled testing on this issue.

I have now done this. The CCRL database is of course connected, any non-connected engines are automatically excluded.

Intel Non-SSE shows as +15 Elo over AMD SSE4. But with error margins +/- 25 Elo it is inconclusive.

Dear Ray,

+1


Btw,i have run a such quick test to see the strength influence difference between SSE and NO SSE:
http://www.sedatcanbaz.com/chess/scct-sse-no-sse/

But however,my testing can not be taken very seriously due to we need more games under different hardwares and opening books
Plus,Critter 1.4 1 core has many games 'forfeits on time' and those time losses can lead to another misunderstandings...

Best,
Sedat

[Modern Times]

Intel Non-SSE shows as +15 Elo over AMD SSE4. But with error margins +/- 25 Elo it is inconclusive.

I should add - inconclusive, but it is the AMD games that are depressing Komodo's rating. Perhaps we should add more. That could improve things for Komodo, or make things worse for it.

But the original discussion here was the 404 blitz list, and they are all Intel games, albeit non-SSE. So AMD is not a factor in the blitz list.

[lkaufman]

Our data for Komodo gave 90 per doubling, which is 1.3% per elo, which means 10% = 13 elo. But of course the other engines also gain something from SSE, maybe 5 elo, so we are back to 8 elo assuming both engines use SSE. But some of the opposing engines don't have SSE capability, so this means the value of SSE for Komodo is between 8 and 13, let's say 10. Yes, there is some decrease in the value of each doubling with greater depth for a given engine, but it seems that as the engines improve the value of doubling has not been shrinking. It all depends on the nature of the improvement. Since the doubling value has been calculated from recent versions where the range of speed is fairly small, I think this factor is negligible for the present calculation.

Hello dear Larry,

Of course i give you right,you and Don are the owners of Komodo...
That's why probably you know better than all about Komodo's Elo speed doubling

Btw,
I have no much info or idea about the rest people hardware speed Elo testings...
But however,i strongly believe that the right Hardware speed Elo measuring method should be done in Auto232 mode

For example,i have some Auto232 Hardware speed Elo testings:
http://www.sedatcanbaz.com/chess/ratings/scct-auto232/

Where i've noticed that any Engine has different Elo performance,in case of doubling the processor speed
Especially Naum 6 core has the highest Elo speed difference:90 Elo
The lowest Elo difference is Fire 2.2 xTreme x64 6c,just 36 Elo
It seems,some critical engines Elo performance suffer with many cores

Code: Select all

Rank  Name                     Elo    +    -  games  score oppo. draws
1 Houdini 2.0c Pro x64 6c      3423   19   18  1008   70%  3261   42% 
5 Houdini 2.0 Pro x64 4c       3363   16   16  1131   61%  3294   49% 
*Elo Difference:60


6 Deep Rybka 4.1 x64 6c         3358   15   15  1363   59%  3305   54%
15 Deep Rybka 4.1 x64 4c        3293   13   14  1602   47%  3314   56% 
*Elo Difference:65

12 Fire 2.2 xTreme x64 6c       3311   23   22   530   52%  3297   63% 
18 Fire 2.2 xTreme x64 4c       3275   16   16  1165   41%  3328   59%
*Elo Difference:36

22 Naum 4.2 x64 6c              3266   22   22   631   50%  3261   46% 
42 Naum 4.2 x64 4c              3176   17   18  1008   35%  3270   45% 
*Elo Difference:90

Houdini 2.0 BenchMarks:

Code: Select all

kN/s    Mate Processor              Speed  Cores L.P  EXE  Hardware User
21416   12s  Intel Core i7 980X     @4.33GHz  6  OFF  x64  Sedat Canbaz
10872   47s  Intel Core i7 920      @3.00GHz  4  OFF  x64  Sedat Canbaz
10336   49s  Intel Extreme QX9650   @3.66GHz  4  OFF  x64  Sedat Canbaz

6 Cores = MP Engines are played on my six core machine (i7 980X @4.33GHz)
4 Cores = MP Engines are played on my Quads (i7 920 @3.0GHz/QX9650 @3.66GHz)


Greetings,
Sedat

I don't think these numbers actually represent a doubling of speed. You can't really go by KNPS when comparing different number of cores. The effective speedup going from 4 cores to 6 cores on even the best scaling MP program is way less than 6 to 4; it's probably something like 5 to 4. So probably your numbers understate the true effect of doubling the processor speed. Still, they are valid for comparing what the different programs get from MP.

Larry

[lkaufman]

Intel Non-SSE shows as +15 Elo over AMD SSE4. But with error margins +/- 25 Elo it is inconclusive.

I should add - inconclusive, but it is the AMD games that are depressing Komodo's rating. Perhaps we should add more. That could improve things for Komodo, or make things worse for it.

But the original discussion here was the 404 blitz list, and they are all Intel games, albeit non-SSE. So AMD is not a factor in the blitz list.

Thanks for doing this. The margin of error on the difference between the two should be about 35, so 15 isn't very meaningful. Our own study didn't show much difference. I'll keep my eyes open for more data on this. We have one thing to check out on our end about this.

[Sedat Canbaz]

I don't think these numbers actually represent a doubling of speed. You can't really go by KNPS when comparing different number of cores. The effective speedup going from 4 cores to 6 cores on even the best scaling MP program is way less than 6 to 4; it's probably something like 5 to 4. So probably your numbers understate the true effect of doubling the processor speed. Still, they are valid for comparing what the different programs get from MP.

Larry

Just my 2 cents more over this issue

Actually I was meaning regarding the MP engines,in case of doubling the processors speed

And as far as i know,there is no any perfect way or ideal Hardware Elo measuring method

But however,from my experience i can say, there are many important factors/influences... where the engines suffer or benefit
*And here are one of them:
AMD/Intel
Single/Multicore
SSE/NON-SSE
Openings
Ponder OFF/ON
32-bit/64-bit
Hashtable size
vs...

In other words,this issue seems to be too complicated and that's why its very hard to measure the real doubling Elo performance
But anyway,i strongly believe that the Engines kns values are quite good indicator about which processors are faster for chess

For example,i have some hardware Elo speed testings,where i noticed the kns values help a lot to measure even the Engines Elo playing strength:


And here is another hardware Elo measuring test:
http://www.sedatcanbaz.com/chess/scct-h ... ournament/



Best,
Sedat

[Modern Times]

Hi Sedat,

Yes it is a very complex issue that is certain.

The results are what they are and it is easy to get too hung up with over-analysing them. Only with tens of thousands of games can you have any real confidence to draw firm conclusions. With reference to the AMD games, they were run by me in a meticulous manner, with a wide variety of different books and opening sets, and all GUI adjudication turned off. So I stand by those games 100%. It is kind of a sense of deja-vu for me though, because I had similarly disappointing results for Komodo 4 on the same machine.