How can I switch off hyperthreading?

[Laskos]

Cutechess-cli was more stable than LittleBlitzer, I managed to have an uninterrupted match:

Code: Select all

   
       Program                                     Score     %     Elo    +   -    Draws

  1 Houdini 3 8-threads                       : 631.5/1200  52.6   3009   13  13   58.6 %
  2 Houdini 3 4-threads                       : 568.5/1200  47.4   2991   13  13   58.6 %

18 +/- 13 points from HT.

[Lavir]

In these last two days I've run all 13 strategical testsuites (1.300 position in total) 10 times (30 secs max for position) to test HT ON vs OFF in my PC with Houdini 3. So I tested in full 13.000 positions for the two setups. One thing to notice is that when I tested without HT I used the BIOS to switch it off so to be certain that only the physical cores were indeed used.

These were the average results:
HT ON: 1255 of 1300 solutions found, avg total time 13:07, avg time for solution: 0.36s
HT OFF: 1251 of 1300 solutions found, avg total time 13:23, avg time for solution: 0.38s

So both the time and solutions found were improved by HT ON with Houdini 3 in my machine.

On important thing IMO is that also if Robert says that the number of solutions found is not so important, in this particular case it is to note that of the 10 runs the minimum solutions found with HT are higher than the maximum solutions found without HT. Now, I think that with 10 runs and 13.000 positions tested it is obvious that, at last with these particular testsuites (that however are not tactical, so they shouldn't benefit from the growing of the tree), with HT more solutions are found and even in less time, usually.

Judging from this test it seems that also in my PC HT is beneficial. The exact amount in ELO I really don't know (I don't think it amounts to much, to be sincere) but the +25% nps count I get with H3 seems to also translate in practice and surely it seems it doesn't do harm (contrary to popular belief).

[Jouni]

BTW haven't you noticed there is 1400 positions in STS?

[Norm Pollock]

BTW haven't you noticed there is 1400 positions in STS?

There are 1397 unique positions in STS. The following positions appear twice each:

Code: Select all

1q2r1k1/1b2bpp1/p2ppn1p/2p5/P3PP1B/2PB1RP1/2P1Q2P/2KR4 b - - bm c4; id "STS(v9.0) Advancement of a/b/c pawns.005"; c0 "c4=10, Bc6=5, Qa7=4, Qa8=5, Qc8=5, b8a8=5";
1q2r1k1/1b2bpp1/p2ppn1p/2p5/P3PP1B/2PB1RP1/2P1Q2P/2KR4 b - - bm c4; id "Undermine.005"; c0 "c4=10, Bc6=7, Qa8=7, Qc8=7";


1r1r1bk1/1bq2p1p/pn2p1p1/2p1P3/5P2/P1NBB3/1P3QPP/R2R2K1 b - - bm Nd5; c0 "Nd5=10, Ba8=8, Kg7=8, a5=9"; id "STS: Knight Outposts/Repositioning/Centralization.004";
1r1r1bk1/1bq2p1p/pn2p1p1/2p1P3/5P2/P1NBB3/1P3QPP/R2R2K1 b - - bm Nd5; id "STS(v12.0) Center Control.008";


r1r3k1/pb3p1p/1pqBp1p1/4P3/3b4/2P2P2/PR1N2PP/2RQ3K w - - bm Ne4; c0 "Ne4=10"; id "STS: Knight Outposts/Repositioning/Centralization.073";
r1r3k1/pb3p1p/1pqBp1p1/4P3/3b4/2P2P2/PR1N2PP/2RQ3K w - - bm Ne4; c0 "Ne4=10, Qe2=3, Rbb1=1, Rbc2=1"; id "STS(v12.0) Center Control.077";

[Lavir]

BTW haven't you noticed there is 1400 positions in STS?

The ones I downloaded are only 1300 in total.

Do you have a link to the full suite?

[Jouni]

Sure

https://sites.google.com/site/strategictestsuite/

[Laskos]

In these last two days I've run all 13 strategical testsuites (1.300 position in total) 10 times (30 secs max for position) to test HT ON vs OFF in my PC with Houdini 3. So I tested in full 13.000 positions for the two setups. One thing to notice is that when I tested without HT I used the BIOS to switch it off so to be certain that only the physical cores were indeed used.

These were the average results:
HT ON: 1255 of 1300 solutions found, avg total time 13:07, avg time for solution: 0.36s
HT OFF: 1251 of 1300 solutions found, avg total time 13:23, avg time for solution: 0.38s

So both the time and solutions found were improved by HT ON with Houdini 3 in my machine.

On important thing IMO is that also if Robert says that the number of solutions found is not so important, in this particular case it is to note that of the 10 runs the minimum solutions found with HT are higher than the maximum solutions found without HT. Now, I think that with 10 runs and 13.000 positions tested it is obvious that, at last with these particular testsuites (that however are not tactical, so they shouldn't benefit from the growing of the tree), with HT more solutions are found and even in less time, usually.

Judging from this test it seems that also in my PC HT is beneficial. The exact amount in ELO I really don't know (I don't think it amounts to much, to be sincere) but the +25% nps count I get with H3 seems to also translate in practice and surely it seems it doesn't do harm (contrary to popular belief).

That's interesting. Some differences from my tests:
I cannot switch off HT in my Bios.
My suites were mostly tactical, but with longer times to solutions and proportionally fewer solutions. STS seemed to me too easy for Houdini. My impression was that HT kicks in after few seconds, judging from NPS. You seem to show that HT is beneficial even at some 1s/move time control, as your time to solutions show.
With 1255/1300 solved, compared to my typical 60% solved, your variance is about 6-7 times smaller, so the difference (1255-1251)/1300 is really several times larger than it seems if compared to 60% tests, and is probably significant. Your time to solutions is some 6% smaller with HT on, comparable to my 10% (with longer times to solutions).

All in all, you seem to show pretty conclusively that HT is beneficial on your PC too.

Kai

[Vinvin]

...
These were the average results:
HT ON: 1255 of 1300 solutions found, avg total time 13:07, avg time for solution: 0.36s
HT OFF: 1251 of 1300 solutions found, avg total time 13:23, avg time for solution: 0.38s

Thanks for your conclusions. In terms of time ration and Elo :
803 sec/787 sec = 1.02 = + 2% speed = around + 2 Elo.

Vincent

[Lavir]

. STS seemed to me too easy for Houdini. My impression was that HT kicks in after few seconds, judging from NPS.

Yes, that's an huge problem, in fact. I think that if the positions were more difficult (so that it did take more time to find them) HT would show even more benefit, but alas I couldn't find a proper positional testsuite apart STS.

The problem of many positional testsuites is that when they are difficult some of the "best" moves in them are not properly best in all scenarios (for example: you can have best move as 1. Ng2, but then the engine wants to play 1. xx then 2.Ng2 with the same exact continuation so it is the same exact thing, but the test is failed) or elsewhere they are too easy (as in the case of STS). The first case always happens in testsuites I have that are targeted for humans, mostly.

If someone more experienced than me can tell me what is one of the best positional testsuites (not so easy as STS) to test engines with I will do the test again with them, so to have a better look at the difference of HT in my PC.

[Lavir]

Thanks for your conclusions. In terms of time ration and Elo :
803 sec/787 sec = 1.02 = + 2% speed = around + 2 Elo.

Vincent

Sure, very little. However, as Kai noticed, since the positions are so easy the result is skewed. H3 never scored below 1249 positions (on the 1300) in every case, and about 90% of those positions are less than 0.5s to find, so the margin of improvemt on the vast majority of them could be very little (even because HT takes a little to "build up").

But, as I said, it is to notice that the maximum positions found without HT (1252) is lower than the minimum of positions found with HT (1253) and in the majority of all ten runs HT ON got 1255 positions and HT OFF 1251 (and the maximum with HT ON was 1257), and it were the positions that took more than 1 sec to find (the remaining 10%) that HT ON did took less, but the results of them get completely obliterated by the mass of the too easy solutions.

So, as a test for ELO difference I don't think this is a good one since 90% of the solutions are too easy for any comparision to be built upon.