Stockfish 301213 - Houdini 4 x64A, 1 CPU Core Test

[mwyoung]

Mark Young wrote:

I have a question. Are you setting up stockfish correctly in your test. When testing stockfish on a 4 core system with the sleeping thread option on. You must set stockfish to 8 cpu not 4. Houdini gains nothing by doing this it stays on 4 cpu. If people are not testing like this they are not testing stockfish at full strength. You stay with robert houdarts instructions on houdini and set to 4 cores. But stockfish runs best using 4 cores and the 4 logical cores. With sleeping threads option turned on.

I am starting to think the Idle Threads Sleep option is not the issue. I went back and read the previous posts and it seems that Mark is testing Houdini vs Stockfish on a hyperthreaded system that has 4 real cores and thus 8 logical threads available. Furthermore it seems that the settings for Houdini have it set to 4 threads while the settings for Stockfish are using 8 threads ? Is that correct ? If so then you are giving Houdini access to only 50% of the system CPU. Just look at the Windows task manager. You need to set both Stockfish and Houdini to the same number of threads. I think you may be misinterpreting what Robert Houdart has said.

Houdini is not able to use logical cores. It is not stockfish. It hurts houdini a bit setting it to 8 cores like stockfish on a 4 core system. Feel free to test this yourself. This is not me alone saying this but robert houdart the programmer of houdini. I posted his answer to this question on this thread. My setup was correct for both houdini and stockfish. I have tested this myself. On ever engine i test. This is why i buy the pro version of chess software when i have only 4 cores. So i can test ever engine to see if ht will help that engine before testing. I want to test every engine on its strongest setting.

Sorry but I think you are totally wrong. If you set Houdini to use 4 threads on a hyperthreaded system which has 4 real cores and 8 logical threads available, then it will only use 50% of the total available cpu. As I said previously, this is confirmed by looking at the Windows task manager. Houdini set at 4 threads uses 50% cpu while Houdini set at 8 threads uses 100% cpu. You are misinterpreting what Robert Houdart has said. If you are testing Houdini set to 4 threads versus any engine set to 8 threads then your test is simply not valid.

Well then congratulations are in order. Nice work . Mystery solved.

[mwyoung]

Mark Young wrote:

I have a question. Are you setting up stockfish correctly in your test. When testing stockfish on a 4 core system with the sleeping thread option on. You must set stockfish to 8 cpu not 4. Houdini gains nothing by doing this it stays on 4 cpu. If people are not testing like this they are not testing stockfish at full strength. You stay with robert houdarts instructions on houdini and set to 4 cores. But stockfish runs best using 4 cores and the 4 logical cores. With sleeping threads option turned on.

I am starting to think the Idle Threads Sleep option is not the issue. I went back and read the previous posts and it seems that Mark is testing Houdini vs Stockfish on a hyperthreaded system that has 4 real cores and thus 8 logical threads available. Furthermore it seems that the settings for Houdini have it set to 4 threads while the settings for Stockfish are using 8 threads ? Is that correct ? If so then you are giving Houdini access to only 50% of the system CPU. Just look at the Windows task manager. You need to set both Stockfish and Houdini to the same number of threads. I think you may be misinterpreting what Robert Houdart has said.

Houdini is not able to use logical cores. It is not stockfish. It hurts houdini a bit setting it to 8 cores like stockfish on a 4 core system. Feel free to test this yourself. This is not me alone saying this but robert houdart the programmer of houdini. I posted his answer to this question on this thread. My setup was correct for both houdini and stockfish. I have tested this myself. On ever engine i test. This is why i buy the pro version of chess software when i have only 4 cores. So i can test ever engine to see if ht will help that engine before testing. I want to test every engine on its strongest setting.

Sorry but I think you are totally wrong. If you set Houdini to use 4 threads on a hyperthreaded system which has 4 real cores and 8 logical threads available, then it will only use 50% of the total available cpu. As I said previously, this is confirmed by looking at the Windows task manager. Houdini set at 4 threads uses 50% cpu while Houdini set at 8 threads uses 100% cpu. You are misinterpreting what Robert Houdart has said. If you are testing Houdini set to 4 threads versus any engine set to 8 threads then your test is simply not valid.

Well then congratulations are in order. Nice work . Mystery solved.

[Paul Bedrey]

This was taken from my Houdini 3 read me file.

Maximum number of threads (cores) used by the analysis.

Default is hardware-dependent, min 1, max 6 (Standard) or 32 (Pro).

Houdini will automatically limit the number of threads to the number of logical processors of your hardware. If your computer supports hyper-threading it is recommended not using more threads than physical cores, as the extra hyper-threads would usually degrade the performance of the engine. See the FAQ about hyper-threading.

I think Mark is correct.

[bnculp]

Yes upon further review it appears that I am "totally wrong". Apologies to Mark. I even did some testing, Houdini 4 threads is better than Houdini 8 threads on a hyperthreaded system. I am going outside to shovel a whole lot of snow. Lol

[mwyoung]

Yes upon further review it appears that I am "totally wrong". Apologies to Mark. I even did some testing, Houdini 4 threads is better than Houdini 8 threads on a hyperthreaded system. I am going outside to shovel a whole lot of snow. Lol

I give you credit for testing your theory, that is more then most have done here in this thread. Keep on testing. Thanks.

[lkaufman]

You are most probably right !
Funny actually, SW developers not knowing the best settings for their own SW.
Though, come to think of it, even YOU don't know the best settings for Komodo for all situations and conditions, in my opinion !

Nor do you, or anyone else. It is impossible to determine best settings for long time control games, because this would require playing thousands of such games, bv which time the version would be obsolete.

[lkaufman]

Yes upon further review it appears that I am "totally wrong". Apologies to Mark. I even did some testing, Houdini 4 threads is better than Houdini 8 threads on a hyperthreaded system. I am going outside to shovel a whole lot of snow. Lol

I give you credit for testing your theory, that is more then most have done here in this thread. Keep on testing. Thanks.

After 849 games the Stockfish lead over Houdini (4 cores each) is now 27 elo. So I don't think that Mark's running 8 threads vs 4 was the reason for the SF win, it appears to be just that SF is much better at MP. since Houdini is still clearly better at bullet single core. Mark, I would recommend that you rerun your match setting BOTH sides at 4 threads, not because I think your result was wrong, but because it is not at all clear to me that using 8 threads helped SF. MP performance degrades a lot going from 4 to 8 cores, and while SF seems to compensate much better than Houdini for this, it may not be enough to be a net win. I doubt that the SF guys have tested this much if at all. Also there will always be a suspicion that running 8 threads vs 4 somehow biases the test, although I don't see much evidence of this.

[ouachita]

it is impossible to determine best settings for long time control games

and largely or entirely true for most STC too

[mwyoung]

Yes upon further review it appears that I am "totally wrong". Apologies to Mark. I even did some testing, Houdini 4 threads is better than Houdini 8 threads on a hyperthreaded system. I am going outside to shovel a whole lot of snow. Lol

I give you credit for testing your theory, that is more then most have done here in this thread. Keep on testing. Thanks.

After 849 games the Stockfish lead over Houdini (4 cores each) is now 27 elo. So I don't think that Mark's running 8 threads vs 4 was the reason for the SF win, it appears to be just that SF is much better at MP. since Houdini is still clearly better at bullet single core. Mark, I would recommend that you rerun your match setting BOTH sides at 4 threads, not because I think your result was wrong, but because it is not at all clear to me that using 8 threads helped SF. MP performance degrades a lot going from 4 to 8 cores, and while SF seems to compensate much better than Houdini for this, it may not be enough to be a net win. I doubt that the SF guys have tested this much if at all. Also there will always be a suspicion that running 8 threads vs 4 somehow biases the test, although I don't see much evidence of this.

I have done a number of test on this, the conclusion from the test is that it does not hurt stockfish at all, and seems to help some. Not a lot, but a few elo. So since the stockfish team said to use Stockfish with 8 threads of a 4 core system for best results. I test what the stockfish team said was best, since I could find no reasons from my pretest to dispute their claim.

So if it does not hurt, and may help why not test this way. If you are right, it does not change any of my results, if the stockfish team is correct. It may give Stockfish a few more deserved elo points in testing.

I am not a promoter of any program, if the way I test stockfish is hurting it by suspicion but is the best and most accurate way to test Stockfish. So be it, I am not here to convince people of anything. Just share my test results, and test the chess programs the most accurate why I know how.

[bnculp]

I am currently running a superfast 1000 game test (15 sec+.05 sec) Stockfish 8-thread vs Stockfish 4-thread on my hyperthreaded system. Idle Threads Sleep is set to default value True. After 160 games Stockfish 8-thread is up by 17 ELO.