Stockfish 020114 - Houdini 4 x64A Testing 39 of 100 played.

[lkaufman]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

I'm not sure this has been established. It is clear that HT on hurts Houdini, but I don't think anyone has determined whether SF plays better with HT on using 8 threads or with HT off using 4. I suspect it is about equal. My conclusions from all of this and my own testing is that Mark's tests show that if you must use HT for whatever reason SF should use 8 threads and is considrably stronger than Houdini on 4 or 8. However if you can turn HT off then they become very close in strength, although I still give a slight edge to SF. On single core Houdini is much stronger regardless of HT (all of this assumes bullet/blitz levels), and is probably slightly stronger on two cores, at least with HT off.

That is why I am testing HT. That is the only way to establish if this is true. My conditions are always posted in my test game results.

I'm not sure what "if this is true" refers to; your test does not tell us anything about whether SF plays better with HT on or with HT off; you would have to actually disable HT and compare times to complete X iterations, averaged over many positions, with the times with HT on and using 8 threads to get the answer. It is possible, maybe even likely, that both engines play worse with HT, but Houdini plays "more worse" (pardon the bad grammar).

[ouachita]

one has to use hyper threading for stockfish while not for Houdini.

"Houdini will automatically limit the number of threads to the number of logical processors of your hardware.

Yea, I'm testing these issues too.

My question is, if HT is enabled in the BIOS, will H actually limit the number of threads to the number of cores or will it covertly HT?

[bnculp]

"Houdini will automatically limit the number of threads to the number of logical processors of your hardware.

Yea, I'm testing these issues too.

My question is, if HT is enabled in the BIOS, will H actually limit the number of threads to the number of cores or will it covertly HT?

logical processors is the key. 16 real cores with HT On means 32 logical processors.

[mwyoung]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

I'm not sure this has been established. It is clear that HT on hurts Houdini, but I don't think anyone has determined whether SF plays better with HT on using 8 threads or with HT off using 4. I suspect it is about equal. My conclusions from all of this and my own testing is that Mark's tests show that if you must use HT for whatever reason SF should use 8 threads and is considrably stronger than Houdini on 4 or 8. However if you can turn HT off then they become very close in strength, although I still give a slight edge to SF. On single core Houdini is much stronger regardless of HT (all of this assumes bullet/blitz levels), and is probably slightly stronger on two cores, at least with HT off.

That is why I am testing HT. That is the only way to establish if this is true. My conditions are always posted in my test game results.

I'm not sure what "if this is true" refers to; your test does not tell us anything about whether SF plays better with HT on or with HT off; you would have to actually disable HT and compare times to complete X iterations, averaged over many positions, with the times with HT on and using 8 threads to get the answer. It is possible, maybe even likely, that both engines play worse with HT, but Houdini plays "more worse" (pardon the bad grammar).

This is why I put the official release Stockfish DD in my test, testing it with default settings, no HT. There are other testers if they wish could test all the possible permutations and versions of Stockfish.

That is not a burning question for myself, I have done this. If others need to test this they can also test. Some are testing this and posting the results on this thread.

You assume I have not looked into this... I could find nothing that shows HT hurts both Stockfish and Houdini, just Houdini worst. I can test this.

And other testers are testing this...

[lkaufman]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

I'm not sure this has been established. It is clear that HT on hurts Houdini, but I don't think anyone has determined whether SF plays better with HT on using 8 threads or with HT off using 4. I suspect it is about equal. My conclusions from all of this and my own testing is that Mark's tests show that if you must use HT for whatever reason SF should use 8 threads and is considrably stronger than Houdini on 4 or 8. However if you can turn HT off then they become very close in strength, although I still give a slight edge to SF. On single core Houdini is much stronger regardless of HT (all of this assumes bullet/blitz levels), and is probably slightly stronger on two cores, at least with HT off.

That is why I am testing HT. That is the only way to establish if this is true. My conditions are always posted in my test game results.

I'm not sure what "if this is true" refers to; your test does not tell us anything about whether SF plays better with HT on or with HT off; you would have to actually disable HT and compare times to complete X iterations, averaged over many positions, with the times with HT on and using 8 threads to get the answer. It is possible, maybe even likely, that both engines play worse with HT, but Houdini plays "more worse" (pardon the bad grammar).

This is why I put the official release Stockfish DD in my test, testing it with default settings, no HT. There are other testers if they wish could test all the possible permutations and versions of Stockfish.

That is not a burning question for myself, I have done this. If others need to test this they can also test. Some are testing this and posting the results on this thread.

You assume I have not looked into this... I could find nothing that shows HT hurts both Stockfish and Houdini, just Houdini worst. I can test this.

And other testers are testing this...

You said you cannot turn off HT on your machine, so you cannot test it. Testing with SF sleeping threads on or off is not at all the same as testing with HT on or off. I assume you have not looked into it based on your own statement that you cannot turn off HT on your machine.

[bnculp]

I think we need to be careful with our terminology. For me HT on/off or enabled/disabled is a system BIOS function. It cant be On or Off for a specific engine at the same time. When it is On, we have the choice through the engine GUI whether or not to use the extra hyperthreads that it provides.

Conventional wisdom in the past says that HT does not help engine performance. So far most of the debate in this thread has concerned Stockfish and Houdini performance with HT on and the various thread options that can be used. I think Larry is also interested in evaluating HT engine performance as a system function. You cant test HT that way in a single engine match or tournament. This is different from what Tom has been referring to when he says "no HT" he means 4 threads out of 8 max on a 4 core system where HT is enabled.

Clear as mud ??!?

[ouachita]

logical processors is the key. 16 real cores with HT On means 32 logical processors.

This issue has turned out to be a challenge to describe in writing. Nevertheless, since my 16 core CPU allows me to enable or disable HT, my assumptions therefore would be:

A. HT enabled:

1. Both SF and H can run on 1 thru 32 threads;

2. Both SF and H are stronger as core usage increases from 2 to 16;

2. SF may or may not be stronger while using threads 17 thru 32 (ongoing testing indicates HT to 32 does not help SF);

3. H will not benefit and will likely be adversely affected from using threads 17 thru 32 (per RH);

B. HT disabled:

1. Both SF and H can run on only 1 thru 16 cores/threads;

Larry can tell me how much of this applies to Kr.

[mwyoung]

Very interesting Mark!
So for a fair match between Houdini 4 and Stockfish on a modern quad core, one has to use hyper threading for stockfish while not for Houdini. In this way the performance of both programs is maximized. If one sets the same conditions for both programs, one of them is going to have an unfair handicap. As simple as that.

I'm not sure this has been established. It is clear that HT on hurts Houdini, but I don't think anyone has determined whether SF plays better with HT on using 8 threads or with HT off using 4. I suspect it is about equal. My conclusions from all of this and my own testing is that Mark's tests show that if you must use HT for whatever reason SF should use 8 threads and is considrably stronger than Houdini on 4 or 8. However if you can turn HT off then they become very close in strength, although I still give a slight edge to SF. On single core Houdini is much stronger regardless of HT (all of this assumes bullet/blitz levels), and is probably slightly stronger on two cores, at least with HT off.

That is why I am testing HT. That is the only way to establish if this is true. My conditions are always posted in my test game results.

I'm not sure what "if this is true" refers to; your test does not tell us anything about whether SF plays better with HT on or with HT off; you would have to actually disable HT and compare times to complete X iterations, averaged over many positions, with the times with HT on and using 8 threads to get the answer. It is possible, maybe even likely, that both engines play worse with HT, but Houdini plays "more worse" (pardon the bad grammar).

This is why I put the official release Stockfish DD in my test, testing it with default settings, no HT. There are other testers if they wish could test all the possible permutations and versions of Stockfish.

That is not a burning question for myself, I have done this. If others need to test this they can also test. Some are testing this and posting the results on this thread.

You assume I have not looked into this... I could find nothing that shows HT hurts both Stockfish and Houdini, just Houdini worst. I can test this.

And other testers are testing this...

You said you cannot turn off HT on your machine, so you cannot test it. Testing with SF sleeping threads on or off is not at all the same as testing with HT on or off. I assume you have not looked into it based on your own statement that you cannot turn off HT on your machine.

I have 7 computers in my house, some amd, some intel. Some with Bios HT option. My engine vs engine computer I keep nothing else on but chess testing. But I can test on any of them. I was referring to my computer I run my test on. To keep the overhead the lowest for testing engine vs engine. This is why my CPU % at rest is almost 0%.

I can do many things on my other computers, like type this response.

Other testers also have many computers at their homes and are posting the results here on your HT Question.

I have run many test on this including running test positions. I get the same NPS, the same time per depth. Except for the normal variation one would expect from testing MP. That why I tested this many times to get a good average.

Larry, I don't know what more I can do or say, but let others post their results.

You need to explain your theory, how a cpu core processing 1 pipe only per core with HT on can cripple a engine, over having HT off.

The CPU is the same either with HT on or off in Bios. ALL the bios HT ON/OFF option does is keep the programs restricted to 1 pipe on the core. But the 2 pipes are still there regardless of the HT setting in the BIOS.

If my results are incorrect and anything is possible that is why we test. INTEL is going to make many customers upset when running their single thread apps on their intel HT CPU. Because that is all INTEL is mostly making in CPU chips.

And if somehow this is true, I am sure we would have read about this from AMD inc.

Sorry I have to disagree, because of my testing. But others are also testing this and posting their results.

[bnculp]

logical processors is the key. 16 real cores with HT On means 32 logical processors.

This issue has turned out to be a challenge to describe in writing. Nevertheless, since my 16 core CPU allows me to enable or disable HT, my assumptions therefore would be:

A. HT enabled:

1. Both SF and H can run on 1 thru 32 threads;

2. Both SF and H are stronger as core usage increases from 2 to 16;

2. SF may or may not be stronger while using threads 17 thru 32 (ongoing testing indicates HT to 32 does not help SF);

3. H will not benefit and will likely be adversely affected from using threads 17 thru 32 (per RH);

B. HT disabled:

1. Both SF and H can run on only 1 thru 16 cores/threads;

Larry can tell me how much of this applies to Kr.

I agree with most although in my testing with HT enabled on a 4 core system, SF 8-threads did edge out SF 4-threads by 7 ELO. One must also be careful about drawing firm conclusions based on small sample sizes. Even a match of 1000 games which I have been using, has error bars about +- 12 ELO. Another variable in your case is 16 real cores and how each engine handles that kind of powerful hardware. There has been some speculation and some evidence (TCEC and Clemens tours both with no HT or HT off) that Stockfish and Komodo perform better than Houdini on higher powered machines. YMMV

[ouachita]

One must also be careful about drawing firm conclusions based on small sample sizes.

If nothing else, we can agree on this (and almost any other sample size too).