Stockfish 020114 - Houdini 4 x64A Testing 39 of 100 played.

[shrapnel]

BINGO! Mystery solved.

Btw, can you show us your fritz benchmark with HT on and 12 threads, just for the hell of it?

Happy to oblige Arjun Bhai

But I rarely use HT on in my online Engine-Engine matches as its really bad for Houdini 4 !
Strangely, it used to benefit Houdini 3 sometimes, but doesn't benefit Houdini 2 or 4 !

[bnculp]

I understand that it is easy to come to this false conclusion. If 4 threads is only allowing Houdini 50% of the CPU then why did Houdini 4 threads beat Houdini 8 threads ? Think about it ! I will run another test with Houdini 8 threads vs Stockfish 8 threads.

So what would constitute a fair test on a HT enabled system ? 8 threads vs 8 threads ? 4 threads vs 4 threads ? It almost seems you are saying that any test on an HT system is BIASED against Houdini. Think about what that says about Houdini

I thought I explained everything properly, but let me further clarify.
My answer in 2 parts :-
(a) Houdini 4 threads beats Houdini with 8 threads BECAUSE of "the increased inefficiency of the parallel alpha-beta search more than
offsets the speed gain obtained with the additional hyper-threads".
(b) Yes, Barry, I'm sorry, but I'm afraid if you insist on enabling ALL threads for Stockfish, the Test will indeed be biased against Houdini whether Houdini uses 4 threads or 8 !! Let me elucidate further.
If Houdini uses 4 threads on a 4-Core HT-enabled System, it will not be getting full resources of the PC ; while if it uses all 8 threads, the 'increased inefficiency of parallel alpha-beta search' factor kicks in !
Really, its a case of Heads, Houdini loses and Tails, Stockfish wins !
I can't be clearer than that !
So, as I mentioned earlier, the solution is to run the Tests using TWO identical Systems, taking care that HT is disabled in the BIOS itself in the System running Houdini, OR else, if its impossible to disable HT in BIOS, run both Stockfish and Houdini with 4 threads each on a System like yours !
Hope I've made things clearer to excellent and dedicated Testers like you and Mark Young !
Regards

LOL. The bottom line is that you are telling me that any test of Houdini vs Stockfish on a Hyperthread system where all threads are enabled is BIASed against Houdini. I do not accept your conclusion. Apparently Stockfish is better at using hyperthreads plain and simple. That's not a BIAS. I have provided test data both with and without hyperthreading. The results speak for themselves. Just imagine the uproar that would result if I suggested that any tests with Houdini on a non HT system are biased and that the only fair test would be a 2 system test.

[mwyoung]

I understand that it is easy to come to this false conclusion. If 4 threads is only allowing Houdini 50% of the CPU then why did Houdini 4 threads beat Houdini 8 threads ? Think about it ! I will run another test with Houdini 8 threads vs Stockfish 8 threads.

So what would constitute a fair test on a HT enabled system ? 8 threads vs 8 threads ? 4 threads vs 4 threads ? It almost seems you are saying that any test on an HT system is BIASED against Houdini. Think about what that says about Houdini

I thought I explained everything properly, but let me further clarify.
My answer in 2 parts :-
(a) Houdini 4 threads beats Houdini with 8 threads BECAUSE of "the increased inefficiency of the parallel alpha-beta search more than
offsets the speed gain obtained with the additional hyper-threads".
(b) Yes, Barry, I'm sorry, but I'm afraid if you insist on enabling ALL threads for Stockfish, the Test will indeed be biased against Houdini whether Houdini uses 4 threads or 8 !! Let me elucidate further.
If Houdini uses 4 threads on a 4-Core HT-enabled System, it will not be getting full resources of the PC ; while if it uses all 8 threads, the 'increased inefficiency of parallel alpha-beta search' factor kicks in !
Really, its a case of Heads, Houdini loses and Tails, Stockfish wins !
I can't be clearer than that !
So, as I mentioned earlier, the solution is to run the Tests using TWO identical Systems, taking care that HT is disabled in the BIOS itself in the System running Houdini, OR else, if its impossible to disable HT in BIOS, run both Stockfish and Houdini with 4 threads each on a System like yours !
Hope I've made things clearer to excellent and dedicated Testers like you and Mark Young !
Regards

LOL. The bottom line is that you are telling me that any test of Houdini vs Stockfish on a Hyperthread system where all threads are enabled is BIASed against Houdini. I do not accept your conclusion. Apparently Stockfish is better at using hyperthreads plain and simple. That's not a BIAS. I have provided test data both with and without hyperthreading. The results speak for themselves. Just imagine the uproar that would result if I suggested that any tests with Houdini on a non HT system are biased and that the only fair test would be a 2 system test.

The people saying this don't understand computers or hardware. If is totally a fair test. You are not keeping Houdini from using 100% of the CPU.

This will be the last time I explain this as I am moving on with testing.

With HT ON and Houdini set to 4CPU on a 4CPU system. Houdini is using 100% of the CPU POWER, but it is only feeding thought 4 of the 8 pipes at a time. It hurts Houdini to use all 8 pipes. It is not designs to use them effectively. NOT STOCKFISH's problem. And Totally fair.

When Stockfish is using all 8 pipes on a 4 core system. Since each core on the CPU is being feed by 2 pipes each, this is LIKE cutting the CPU MHZ in half, because the CPU has to work on 2 PIPES.

THE CPU % on the task manager is talking about how busy the pipes are in the system. So anything that shows 50% on the task manager that program is using 100% CPU power, but no HT. After 50% You are not gaining real cores, but virtual or logical cores. Meaning now each cores is doing twice the work. So it is like cutting the MHZ in half for each core when using HT.

You are getting nothing for free here........ Just a different way to process, For Houdini using 1 pipe on each core is best. For Stockfish using 2 pipes on each core is best. But a CORE IS A CORE, it is not a real cpu, it is a logical CPU.........................................................................

[mwyoung]

http://www.intel.com/content/www/us/en/ ... video.html

And links to how HT works.

[mohzus]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

[lkaufman]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

I'm not sure this has been established. It is clear that HT on hurts Houdini, but I don't think anyone has determined whether SF plays better with HT on using 8 threads or with HT off using 4. I suspect it is about equal. My conclusions from all of this and my own testing is that Mark's tests show that if you must use HT for whatever reason SF should use 8 threads and is considrably stronger than Houdini on 4 or 8. However if you can turn HT off then they become very close in strength, although I still give a slight edge to SF. On single core Houdini is much stronger regardless of HT (all of this assumes bullet/blitz levels), and is probably slightly stronger on two cores, at least with HT off.

[mwyoung]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

Yes on a intel only cpu, AMD does not have HT.

Why would I dumb down Stockfish on my i7 cpu. It is a feature that stockfish can take avantage of on a Intel CPU (HT). If you have a AMD cpu then I guess Houdini is best at these time controls.

And I did not dumb down Houdini by not using table base, when Stockfish was not able to use table bases.

And anyone could test Houdini 4 with HT, it can use them, but it performs worst. So I could to be "FAIR" test Houdini 4 using 4 real cores, and 4 logical cores. The same as Stockfish it that makes them feel better, but Houdini 4 gains nothing and seem to hurt Houdini a bit.

The programmer for Houdini agrees Robert Houdart, and says for Houdini 4 to perform best, set threads to the number of cores..........

[mwyoung]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

I'm not sure this has been established. It is clear that HT on hurts Houdini, but I don't think anyone has determined whether SF plays better with HT on using 8 threads or with HT off using 4. I suspect it is about equal. My conclusions from all of this and my own testing is that Mark's tests show that if you must use HT for whatever reason SF should use 8 threads and is considrably stronger than Houdini on 4 or 8. However if you can turn HT off then they become very close in strength, although I still give a slight edge to SF. On single core Houdini is much stronger regardless of HT (all of this assumes bullet/blitz levels), and is probably slightly stronger on two cores, at least with HT off.

That is why I am testing HT. That is the only way to establish if this is true. My conditions are always posted in my test game results.

[ouachita]

one has to use hyper threading for stockfish while not for Houdini.

Houdini 4 User's Manual

"Houdini will automatically limit the number of threads to the number of logical processors of your hardware. If your computer supports hyper-threading it is recommended not using more threads than physical cores, as the extra hyper-threads would usually degrade the performance of the engine."

If this is literally the case, a 16 core machine can run with HT enabled in the BIOS and SF set on 32 threads and H4 on 16 threads/cores with HT enabled and both engines might then be at their best, assuming that SF indeed benefits from HT and other assumptions.

HT is not "on" or "off", but rather "enabled" or "disabled", at least on an E5-2687W.

[bnculp]

one has to use hyper threading for stockfish while not for Houdini.

Houdini 4 User's Manual

"Houdini will automatically limit the number of threads to the number of logical processors of your hardware. If your computer supports hyper-threading it is recommended not using more threads than physical cores, as the extra hyper-threads would usually degrade the performance of the engine."

If this is literally the case, a 16 core machine can run with HT enabled in the BIOS and SF set on 32 threads and H4 on 16 threads/cores with HT enabled and both engines might then be at their best, assuming that SF indeed benefits from HT and other assumptions.

HT is not "on" or "off", but rather "enabled" or "disabled", at least on an E5-2687W.

Been there done that. I already posted the equivalent test on my 4 core i7-3720QM with HT On and 8 threads for SF and 4 threads for Houdini. It was that specific test which caused some to claim the test was BIASed. I will post 2 more tests,8 thread vs 8 threads and 4 vs 4, when they finish later today.

Previously posted test results :

Code: Select all

System    Hyperthread       Engine     Threads     ELO 

  i7-3720QM   Yes           SF-311213       8       +7 
  i7-3720QM   Yes           Houdini-4       4       -7 

  i7-3720QM   Yes           SF-311213       8       +7 
  i7-3720QM   Yes           SF-311213       4       -7 

  i7-3720QM   Yes           Houdini-4       8       -14 
  i7-3720QM   Yes           Houdini-4       4       +14 

  i7-2600k    No            SF-311213       4       -16 
  i7-2600k    No            Houdini-4       4       +16 

  i7-2600k    No            SF-311213       1       -53 
  i7-2600k    No            Houdini-4       1       +53 

  i7-2600k    Yes           SF-311213       1       -63 
  i7-2600k    Yes           Houdini-4       1       +63 

  i7-2600k    Yes           SF-311213       2       -19 
  i7-2600k    Yes           Houdini-4       2       +19