The BusyFEN benchmark

[sje]

The BusyFEN benchmark is a speed test of a program's basic move generation, move execution, and move retraction routines.

The idea is to be able to compare different platforms with one version of a program or to compare multiple versions of a program on the same platform, and to do these comparisons independent of any code specific to search, evaluation, and transposition details.

The idea is simple. A program runs the BusyFEN benchmark by calculating the perft(5) of the BusyFEN position:
[d]
Which is 164,075,551. The calculation is performed with no transposition assistance and no bulk counting. All position database items are updated/downdated with each execution/retraction.

After the calculation, the program sums the CPU time needed for all threads used and then reports the node frequency (NPS) and the mean node time.

Some results from Symbolic:

Code: Select all

gadget     Throughput: 113.792 KHz   8.78797 us (400 MHz 32 bit PowerPC)
wendy      Throughput: 152.650 KHz   6.55095 us (700 MHz 32 bit Raspberry Pi)
karen      Throughput: 279.532 KHz   3.57741 us (1 GHz 32 bit BeagleBone Black)
serra      Throughput: 11.4540 MHz   87.3060 ns (3.1 GHz 64 bit quad core AMD Athlon)
kristen    Throughput: 21.2806 MHz   46.9912 ns (3.4 GHz 64 bit quad core Intel Core i7)

[hgm]

The BusyFEN benchmark is a speed test of a program's basic move generation, move execution, and move retraction routines.

This is not really a relevant metric for engine speed, right? No engine I know makes and unmakes moves that it doesn't want to search, just for fun. They all end in move generation, (be it capture-only or full width), in order to conclude that this produced no moves they want to consider anymore, and then return without making and unmaking all the moves they dd not want to consider.

Normal engines do as many MoveGens as MakeMoves. (When they can stand pat, futility pruning would also prune the MakeMove.) Your test does about 40 times as many MakeMoves as MoveGens.

[sje]

This is not really a relevant metric for engine speed, right?

There is no claim that the benchmark is a metric for engine speed.

[hgm]

So why use this for anything, rather than, say, a program to calculate pi? Or just a loop to fill a 256MB array in strides of 4097 (wrapping), and then read it back again? Then you could also compare the same program on different platforms, or make multiple versions and compare them on the same platform?

[sje]

...and to do these comparisons independent of any code specific to search, evaluation, and transposition details.

[hgm]

Well, calculating pi would be pretty independent of any search, evaluation and transposition details, right?