Houdini Scaling on CEGT Rating List

[Milton]

I was looking at the CEGT 40/20 rating list and was surprised to see that Houdini 6 doesn't seem to scale well with extra CPU's on that list:

Houdini 6.0 x64 8 CPU rating 3442
Houdini 6.0 x64 4 CPU rating 3437
A 5 pt. increase.

By comparison:

Stockfish 9.0 x64 8 CPU rating 3506
Stockfish 9.0 x64 4 CPU rating 3471

A 35 pt. increase.

Has anyone else done testing to confirm Houdini's poor scaling, or is this a margin-of-error issue?

[Laskos]

I was looking at the CEGT 40/20 rating list and was surprised to see that Houdini 6 doesn't seem to scale well with extra CPU's on that list:

Houdini 6.0 x64 8 CPU rating 3442
Houdini 6.0 x64 4 CPU rating 3437
A 5 pt. increase.

By comparison:

Stockfish 9.0 x64 8 CPU rating 3506
Stockfish 9.0 x64 4 CPU rating 3471

A 35 pt. increase.

Has anyone else done testing to confirm Houdini's poor scaling, or is this a margin-of-error issue?

Sure error margin issue and maybe pool of opponents issue. We saw Houdini 6 in TCEC and WCCC on 40+ cores, it scales in the same league with the best, SF, Komodo, etc.

[Guenther]

I was looking at the CEGT 40/20 rating list and was surprised to see that Houdini 6 doesn't seem to scale well with extra CPU's on that list:

Houdini 6.0 x64 8 CPU rating 3442
Houdini 6.0 x64 4 CPU rating 3437
A 5 pt. increase.

By comparison:

Stockfish 9.0 x64 8 CPU rating 3506
Stockfish 9.0 x64 4 CPU rating 3471

A 35 pt. increase.

Has anyone else done testing to confirm Houdini's poor scaling, or is this a margin-of-error issue?

If you would post the error margin we could tell more w/o extra work
Have you compared to CCRL too?

[Werner]

Hi,
of course, 370 games do not show too much. But I have had some troubles with Houdini after a Windows update: it was not longer registered.

We will check this on the new pc of our tester. Thanks for the posting.
best wishes
Werner

[Milton]

I was looking at the CEGT 40/20 rating list and was surprised to see that Houdini 6 doesn't seem to scale well with extra CPU's on that list:

Houdini 6.0 x64 8 CPU rating 3442
Houdini 6.0 x64 4 CPU rating 3437
A 5 pt. increase.

By comparison:

Stockfish 9.0 x64 8 CPU rating 3506
Stockfish 9.0 x64 4 CPU rating 3471

A 35 pt. increase.

Has anyone else done testing to confirm Houdini's poor scaling, or is this a margin-of-error issue?

If you would post the error margin we could tell more w/o extra work
Have you compared to CCRL too?

I have looked at the CCRL 40/40 rating list.
As near as I can tell, all Houdini testing was done with 4 CPUs, so this list does not have comparative data about how well Houdini scales with additional CPUs.

[Werner]

I can help with our 40/4 list:

Houdini 6.0 x64 12CPU 3592
Houdini 6.0 x64 4CPU 3470
Houdini 6.0 x64 2CPU 3435
Houdini 6.0 x64 1CPU 3372

[Leo]

Houdini might go the way of Rybka and become a dead engine.

[CMCanavessi]

Houdini might go the way of Rybka and become a dead engine.

At least rybka has a son, Fritz 16

[Graham Banks]

I have looked at the CCRL 40/40 rating list.
As near as I can tell, all Houdini testing was done with 4 CPUs, so this list does not have comparative data about how well Houdini scales with additional CPUs.

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

[Milton]

I have looked at the CCRL 40/40 rating list.
As near as I can tell, all Houdini testing was done with 4 CPUs, so this list does not have comparative data about how well Houdini scales with additional CPUs.

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

http://www.computerchess.org.uk/ccrl/40 ... ons_only=1

Thanks for pointing me to the single cpu list Graham:

So comparing the same programs on the CCRL lists as I did on the CEGT list:


Stockfish 9 64-bit 4CPU Rating: 3432
Stockfish 9 64-bit 1cpu Rating: 3367

A difference of 65 points.

Houdini 6 64-bit 4CPU Rating: 3396
Houdini 6 64-bit 1CPU Rating: 3336

A difference of 60 points.

So the scaling of the two programs from 1 CPU to 4 CPUs in CCRL are very close.