FGRL 60 min + 15 sec Rating list - Komodo 10.4

[Jouni]

Not so sure: CEGT 40/20 4CPU gives 58.3 % for SF8!

[JJJ]

I m not sure either. I was for the time control.

[beram]

I m not sure either. I was for the time control.

Time control maybe yes, more cores sure no
See CEGT 40/4 http://www.cegt.net/40_4_Ratinglist/40_ ... liste.html

Komodo 10.3 1 cpu 42% 200 games
Komodo 10.3 2 cpu 39,5% 100 games
Komodo 10.3 12 cpu 37,5% 100 games

[mjlef]

I m not sure either. I was for the time control.

Time control maybe yes, more cores sure no
See CEGT 40/4 http://www.cegt.net/40_4_Ratinglist/40_ ... liste.html

Komodo 10.3 1 cpu 42% 200 games
Komodo 10.3 2 cpu 39,5% 100 games
Komodo 10.3 12 cpu 37,5% 100 games

Trying to draw conclusions from 100-200 game sets is not going to work unless the programs have a huge rating difference between them. Larry is basing his claims on a lot more data.

Some good public data is here:

http://www.fastgm.de/threads2.html

The runs for Komodo 9.3 show Komodo 9.3 going increasing 293 elo in a match of 1 versus 16 threads. This is a pretty meaningful 3000 game match. Stockfish 7 (I believe the first official release with the improve Lazy SMP search) increasing 228 elo when going from 1 to 16 threads. Again 3000 games all at 60+0.05. It is possible that the Stockfish guys have improved thread/core scaling more since then, but I have not seen anything in statistically significant results to suggest a 64 elo improvement in 16 core scaling. We have also improved Komodo's MP search since these runs. It would be great of Andreas could repeat these runs with Komodo 10.4 and Stockfish 8, just to see what the scaling is now.

Also in the same data, Stockfish 6 (before Lazy SMP) increased 166 elo going from 1 to 16 cores. So Lazy SMP helped Stockfish a great deal on lots of cores.

Anyone with data for more than 4 cores, please post.

Mark

[beram]

I m not sure either. I was for the time control.

Time control maybe yes, more cores sure no
See CEGT 40/4 http://www.cegt.net/40_4_Ratinglist/40_ ... liste.html

Komodo 10.3 1 cpu 42% 200 games
Komodo 10.3 2 cpu 39,5% 100 games
Komodo 10.3 12 cpu 37,5% 100 games

Trying to draw conclusions from 100-200 game sets is not going to work unless the programs have a huge rating difference between them. Larry is basing his claims on a lot more data.

Some good public data is here:

http://www.fastgm.de/threads2.html

The runs for Komodo 9.3 show Komodo 9.3 going increasing 293 elo in a match of 1 versus 16 threads. This is a pretty meaningful 3000 game match. Stockfish 7 (I believe the first official release with the improve Lazy SMP search) increasing 228 elo when going from 1 to 16 threads. Again 3000 games all at 60+0.05. It is possible that the Stockfish guys have improved thread/core scaling more since then, but I have not seen anything in statistically significant results to suggest a 64 elo improvement in 16 core scaling. We have also improved Komodo's MP search since these runs. It would be great of Andreas could repeat these runs with Komodo 10.4 and Stockfish 8, just to see what the scaling is now.


Also in the same data, Stockfish 6 (before Lazy SMP) increased 166 elo going from 1 to 16 cores. So Lazy SMP helped Stockfish a great deal on lots of cores.

Anyone with data for more than 4 cores, please post.

Mark

Would be nice IF Larry would show its data on which he based that claim or should I say remark

[Milos]

Scaling with more time and scaling with more cores/threads are two different issues, though the first will help the second. I think it is clearer that Komodo scales better than Stockfish with more cores than with more time.

Regarding more time.
SF8:K is constant 55% at all TCs, therefore there is no better scaling at all. Actually since the gap doesn't reduce due to diminishing return it is more probable that SF has better scaling with TC.
The change in Elo is just an artefact of K's contempt that SF doesn't have.
At very short TC contempt effect is non-existent since draw rate is really low and SF has a larger lead. At longer TC draw rate increases and K's contempt helps reducing the Elo gap. However, that is totally artificial.
So please, stop producing bogus claims.
Regarding more cores your claims are even less believable.

[Milos]

Some good public data is here:

http://www.fastgm.de/threads2.html

The runs for Komodo 9.3 show Komodo 9.3 going increasing 293 elo in a match of 1 versus 16 threads. This is a pretty meaningful 3000 game match. Stockfish 7 (I believe the first official release with the improve Lazy SMP search) increasing 228 elo when going from 1 to 16 threads. Again 3000 games all at 60+0.05. It is possible that the Stockfish guys have improved thread/core scaling more since then, but I have not seen anything in statistically significant results to suggest a 64 elo improvement in 16 core scaling. We have also improved Komodo's MP search since these runs. It would be great of Andreas could repeat these runs with Komodo 10.4 and Stockfish 8, just to see what the scaling is now.

Also in the same data, Stockfish 6 (before Lazy SMP) increased 166 elo going from 1 to 16 cores. So Lazy SMP helped Stockfish a great deal on lots of cores.

Anyone with data for more than 4 cores, please post.

You are comparing apples and oranges. You don't have a consistent metric therefore your results are meaningless.
The only reliable metric for SMP is to measure efficiency i.e. to provide time control multiplier at which single core K is equal in performance to N core K.
We already know very reliably SF8's efficiency is 95.5%. However there is no clue about it for K.

[JJJ]

Scaling with more time and scaling with more cores/threads are two different issues, though the first will help the second. I think it is clearer that Komodo scales better than Stockfish with more cores than with more time.

Regarding more time.
SF8:K is constant 55% at all TCs, therefore there is no better scaling at all. Actually since the gap doesn't reduce due to diminishing return it is more probable that SF has better scaling with TC.
The change in Elo is just an artefact of K's contempt that SF doesn't have.
At very short TC contempt effect is non-existent since draw rate is really low and SF has a larger lead. At longer TC draw rate increases and K's contempt helps reducing the Elo gap. However, that is totally artificial.
So please, stop producing bogus claims.
Regarding more cores your claims are even less believable.

Good point.

[Milos]

Scaling with more time and scaling with more cores/threads are two different issues, though the first will help the second. I think it is clearer that Komodo scales better than Stockfish with more cores than with more time.

Regarding more time.
SF8:K is constant 55% at all TCs, therefore there is no better scaling at all. Actually since the gap doesn't reduce due to diminishing return it is more probable that SF has better scaling with TC.
The change in Elo is just an artefact of K's contempt that SF doesn't have.
At very short TC contempt effect is non-existent since draw rate is really low and SF has a larger lead. At longer TC draw rate increases and K's contempt helps reducing the Elo gap. However, that is totally artificial.
So please, stop producing bogus claims.
Regarding more cores your claims are even less believable.

Good point.

This can also be clearly seen from results. At 60s+0.6s draw rates for SF, K and H are respectively 15.65%, 15.08% and 15.13%. At 60m+15s draw rates for SF, K and H are respectively 53.85%, 49.63% and 57.1%.
At 60m+15s contempt really kicks in.

[mjlef]

Some good public data is here:

http://www.fastgm.de/threads2.html

The runs for Komodo 9.3 show Komodo 9.3 going increasing 293 elo in a match of 1 versus 16 threads. This is a pretty meaningful 3000 game match. Stockfish 7 (I believe the first official release with the improve Lazy SMP search) increasing 228 elo when going from 1 to 16 threads. Again 3000 games all at 60+0.05. It is possible that the Stockfish guys have improved thread/core scaling more since then, but I have not seen anything in statistically significant results to suggest a 64 elo improvement in 16 core scaling. We have also improved Komodo's MP search since these runs. It would be great of Andreas could repeat these runs with Komodo 10.4 and Stockfish 8, just to see what the scaling is now.

Also in the same data, Stockfish 6 (before Lazy SMP) increased 166 elo going from 1 to 16 cores. So Lazy SMP helped Stockfish a great deal on lots of cores.

Anyone with data for more than 4 cores, please post.

You are comparing apples and oranges. You don't have a consistent metric therefore your results are meaningless.
The only reliable metric for SMP is to measure efficiency i.e. to provide time control multiplier at which single core K is equal in performance to N core K.
We already know very reliably SF8's efficiency is 95.5%. However there is no clue about it for K.

95.5% efficiency is nonsense. I quoted matches with 3000 games where Komodo increased elo more than Stockfish (or similar generations of each program). Komodo's MP search is not designed to get "more efficient". It is to get more elo. That is what we say when we claim Komodo scales well (its elo increases more with time and threads). We have never made any claims about "efficiency".

Larry's more recent data still shows what I wrote to be true, but I would love a more comprehensive run wilt lots of games and varying threads on suitable hardware. It is possible something in Stockfish's MP has changed to make it scale better, but I have not seen any conclusive data on that.

These threads just seem to wonder away from the point.