Crafty on Cray Blitz vs. Rybka3

[mhull]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

Remember that this is not exactly 10:1. Given the choice of a 10ghz processor, or 10 1ghz processors, I would take the single 10ghz every time. Crafty loses about 30% for each additional processor. So four really acts like three (this is probably true of most SMP programs). It is not as easy to compare this way. If I compare I either always use single-cpu for both, or multiple cpu for both. Then the speed comparisons make sense. Otherwise the numbers are off by a significant amount (25% in this case).

But Lance has said previously that "Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP" and this is how he had calculated that Crafty had a 5:1 advantage in the previous match. I assume the same method (a benchmark) was used in this case too, and it is really a 10:1 advantage.

No it isn't. Crafty's NPS scaling is very good. But parallel search efficiency does not scale as well as NPS for _any_ program. 4 cpus at X gigahertz is roughly the same as one cpu at 3X gigahertz. Even though the NPS will be almost 4x faster. But the trees searched will be 33% larger.

As I said, it is closer to 7x faster...

Then should time-to-ply be the better metric here?

[bob]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

Remember that this is not exactly 10:1. Given the choice of a 10ghz processor, or 10 1ghz processors, I would take the single 10ghz every time. Crafty loses about 30% for each additional processor. So four really acts like three (this is probably true of most SMP programs). It is not as easy to compare this way. If I compare I either always use single-cpu for both, or multiple cpu for both. Then the speed comparisons make sense. Otherwise the numbers are off by a significant amount (25% in this case).

But Lance has said previously that "Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP" and this is how he had calculated that Crafty had a 5:1 advantage in the previous match. I assume the same method (a benchmark) was used in this case too, and it is really a 10:1 advantage.

No it isn't. Crafty's NPS scaling is very good. But parallel search efficiency does not scale as well as NPS for _any_ program. 4 cpus at X gigahertz is roughly the same as one cpu at 3X gigahertz. Even though the NPS will be almost 4x faster. But the trees searched will be 33% larger.

As I said, it is closer to 7x faster...

Then should time-to-ply be the better metric here?

Good question. You have to be careful with that metric as well, as it will require a bunch of test positions so that you get an accurate "average". There are positions (kopec 22, Bxe4) where Crafty regularly produces a "super-linear speedup" (>4 on 4 cpus). And there are some where it produces a very poor speedup (1.5 on a 4 cpu system). So an average is needed. Hence my formula "speedup = 1 + (NCPUS -1) * 0.7... that is an average over a bunch of test positions, including the Cray Blitz positions where they are not tactical positions specifically, just positions from a real game.

NPS is a poor indicator of anything, other than if NPS doesn't scale by 4.0 on 4 cpus, that will certainly limit the parallel search speedup even fruther. For example, if you could find a 4-cpu system where Crafty's NPS only scales 2x, then the speedup would be su = (1 + (3) * .7) * .5 or about 1.55x rather than 3.1x. Most of the time Crafty's NPS scales just fine so I don't include that in the formula, but it is there...

[Uri Blass]

This time, I have a 10:1 hardware advantage for Crafty.

Code: Select all

              Score     123456789012345678901234567890
------------------------------------------------------
 1: Rybka   22.5 / 30   0111==1101=11111=1=1=1=101=11=
 2: Crafty   7.5 / 30   1000==0010=00000=0=0=0=010=00=
------------------------------------------------------

Rybka 2.2 (free version) ran on an old Vaio Notebook (P3M) while Crafty 22.1 ran on a quadcore HP Optiplex (Q6600) using all cores (4 threads).

Time control is 15+2, on 15 Nunn positions.

Remember that this is not exactly 10:1. Given the choice of a 10ghz processor, or 10 1ghz processors, I would take the single 10ghz every time. Crafty loses about 30% for each additional processor. So four really acts like three (this is probably true of most SMP programs). It is not as easy to compare this way. If I compare I either always use single-cpu for both, or multiple cpu for both. Then the speed comparisons make sense. Otherwise the numbers are off by a significant amount (25% in this case).

But Lance has said previously that "Crafty bench shows 0.45M nps on the Vaio, and 2.35M nps (2 threads) on the HP" and this is how he had calculated that Crafty had a 5:1 advantage in the previous match. I assume the same method (a benchmark) was used in this case too, and it is really a 10:1 advantage.

Note that 2.35:0.45 is slightly bigger than 5:1 so Bob can correct the 7:1 estimate to 7*2.35/2.25:1 that is near 7.3:1

Uri