New computer suggestions please

[AndrewGrant]

I read a handful of threads by the people contributing to stockfish dev. The consensus was that hyper threading can be trusted in high end I7's, and I9, and any ryzen.

Where did your old cpu fall? High end I5?

[AlvaroBegue]

Here you go (I made the number of threads a 4th argument):

Code: Select all

alvaro@ryzen7:~/ruy$ python Benchmark.py ruy uci 13 1
Running benchmark for ruy
time is 7562.18862534ms
Average time is 7562.18862534ms
alvaro@ryzen7:~/ruy$ python Benchmark.py ruy uci 13 7
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
time is 8483.46877098ms
time is 8613.10839653ms
time is 8885.61081886ms
time is 9016.95370674ms
time is 9112.26940155ms
time is 9159.39188004ms
time is 9395.42651176ms
Average time is 8952.31849807ms
alvaro@ryzen7:~/ruy$ python Benchmark.py ruy uci 13 8
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
time is 8785.17913818ms
time is 8822.55339622ms
time is 8784.897089ms
time is 8910.9582901ms
time is 8865.31710625ms
time is 8880.64718246ms
time is 9045.70007324ms
time is 9435.72092056ms
Average time is 8941.3716495ms
alvaro@ryzen7:~/ruy$ python Benchmark.py ruy uci 13 14
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
time is 11946.5956688ms
time is 11770.2217102ms
time is 11803.920269ms
time is 11778.6698341ms
time is 12179.7654629ms
time is 12280.1966667ms
time is 12253.9894581ms
time is 12551.0938168ms
time is 12494.2867756ms
time is 12538.6936665ms
time is 12716.8354988ms
time is 12722.933054ms
time is 12817.3177242ms
time is 12788.2659435ms
Average time is 12331.6275392ms
alvaro@ryzen7:~/ruy$ python Benchmark.py ruy uci 13 15
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
time is 12201.8864155ms
time is 12163.120985ms
time is 12346.4756012ms
time is 12271.582365ms
time is 12382.2724819ms
time is 12630.096674ms
time is 12755.5975914ms
time is 12748.8539219ms
time is 13027.1704197ms
time is 13224.6105671ms
time is 13275.2289772ms
time is 13336.6122246ms
time is 13365.3738499ms
time is 13450.9942532ms
time is 13404.8964977ms
Average time is 12838.984855ms
alvaro@ryzen7:~/ruy$ python Benchmark.py ruy uci 13 16
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
Running benchmark for ruy
time is 13330.6000233ms
time is 13620.9049225ms
time is 14005.8543682ms
time is 13987.3065948ms
time is 14041.1689281ms
time is 14400.8395672ms
time is 14860.6822491ms
time is 14866.0817146ms
time is 14860.490799ms
time is 14879.0998459ms
time is 14871.3872433ms
time is 14913.4449959ms
time is 14965.1880264ms
time is 15073.1227398ms
time is 15151.4678001ms
time is 15154.4811726ms
Average time is 14561.3825619ms

Since you don't know how RuyDos behaves, perhaps the Stockfish 8 numbers are more informative:

Code: Select all

alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 13 1
Running benchmark for stockfish
time is 1927.50883102ms
Average time is 1927.50883102ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 13 7
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 2189.59736824ms
time is 2304.13413048ms
time is 2300.41098595ms
time is 2294.51799393ms
time is 2320.76406479ms
time is 2337.651968ms
time is 2330.07478714ms
Average time is 2296.73589979ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 13 8
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 2255.08618355ms
time is 2280.06339073ms
time is 2288.20562363ms
time is 2297.22118378ms
time is 2330.30748367ms
time is 2342.74482727ms
time is 2337.25309372ms
time is 2355.89957237ms
Average time is 2310.84766984ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 13 14
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 2476.4251709ms
time is 2591.08257294ms
time is 2667.02771187ms
time is 2681.19359016ms
time is 2707.31163025ms
time is 2708.89425278ms
time is 2705.06429672ms
time is 2712.2900486ms
time is 2782.10330009ms
time is 2786.6165638ms
time is 2915.46440125ms
time is 2929.63624001ms
time is 2953.16457748ms
time is 3010.61534882ms
Average time is 2759.0635504ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 13 15
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 2475.69012642ms
time is 2529.98185158ms
time is 2613.76166344ms
time is 2641.45183563ms
time is 2660.44688225ms
time is 2785.53795815ms
time is 2809.40532684ms
time is 2867.44737625ms
time is 2855.57103157ms
time is 2858.09135437ms
time is 2890.61260223ms
time is 3021.93117142ms
time is 3055.48334122ms
time is 3092.39435196ms
time is 3104.35390472ms
Average time is 2817.4773852ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 13 16
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 2635.80584526ms
time is 2874.08518791ms
time is 2915.36092758ms
time is 2944.27204132ms
time is 2961.64774895ms
time is 2985.47816277ms
time is 2991.751194ms
time is 2996.91963196ms
time is 3040.8039093ms
time is 3064.94784355ms
time is 3104.94494438ms
time is 3159.86824036ms
time is 3145.59698105ms
time is 3180.56583405ms
time is 3205.98530769ms
time is 3194.1382885ms
Average time is 3025.13575554ms

[jdart]

I think their conclusion is that you can get some performance boost by running Stockfish with hyperthreading on and number of threads > number of physical cores.

This does not mean you will get the same performance on 32 cores with hyperthreading that you'd get on a system with 32 physical cores.

It also likely doesn't mean that you can run twice the workload (with hyperthreading enabled, vs. a system without it) for other tasks such as a match that is using each core to run a different engine. You will get some resource contention (actually, I am guessing, quite a lot).

Personally, I just disable hyperthreading on my test machines.

--Jon

[AndrewGrant]

Those numbers are quite surprising to me. Even before hyper threading is needed the numbers fluctuate greatly.

with those numbers I would not even trust running on 7 cores at once.

Did you have a bunch of other programs running at the same time?

Do you have slow memory read/writes? you could add a line to the uci part to tell stockfish to use just 1MB hash.

Try running with a higher depth for stockfish. Maybe there is a lot of blocking when allocating tables on startup.

Code: Select all

process.stdin.write("setoption name Hash value 1\n")

[AndrewGrant]

Did some maths with the data you gave

Stockfish Runs

Code: Select all

    #Threads    Avg Time   LowerErr%   UpperErr%
           1      1928.0         0.0         0.0
           7      2297.0         4.7         1.8
           8      2311.0         2.4         1.9
          14      2759.0        10.2         9.1
          15      2817.0        12.1        10.2
          16      3025.0        12.9         6.0

RuyDos Runs

Code: Select all

    #Threads    Avg Time   LowerErr%   UpperErr%
           1      7562.0         0.0         0.0
           7      8952.0         5.2         4.9
           8      8941.0         1.8         5.5
          14     12332.0         4.6         3.9
          15     12839.0         5.3         4.8
          16     14561.0         8.5         4.1

So RuyDos has much less differences in the completion speeds between each thread, which I imagine is not a characteristic of RuyDos, but because your engine took longer to run the benchmark positions.

So this does not really answer your question of whether or not the hyper threaded cores will work for testing. (RuyDosDev vs RoyDosBase getting even cpu time), but we have shown (with a sample size of 1...) that if you run 14threads over 7threads, the 14threads have the same variance of running times as the 7threads (just about).

Although, the 14threads runs at 72.5% the speed of 7threads, so you would need to increase your TC by 37.7%.

Which works out to be a 45% increase in effective CPU time, so you can work through 3 tests in the time it takes 7threads to do 2.

If you want to take any of this as an exact science, I would recommend running much higher bench depths, since my python program may me messing with the data as well.

In regards to my last post : It seems like you cannot avoid the 5% time margins. My margins are around .125%, but maybe that's just because I have so few cores. If you are happy running on 7threads, I see no reason to not be happy with 14threads. Maybe 12threads if you want to be extra safe, 1 core for Cutechess, and 1 for OS

[AlvaroBegue]

Thanks, Andrew. I have to think about these numbers a bit and perhaps run more tests.

Here's the result of running Stockfish with depth 16, so it takes a more measurable amount of time:

Code: Select all

alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 16 1
Running benchmark for stockfish
time is 8467.26298332ms
Average time is 8467.26298332ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 16 7
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 9423.8421917ms
time is 9420.59993744ms
time is 9437.96086311ms
time is 9455.54900169ms
time is 9466.65668488ms
time is 9456.16531372ms
time is 9494.90451813ms
Average time is 9450.81121581ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 16 8
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 9418.76316071ms
time is 9560.08338928ms
time is 9577.58426666ms
time is 9625.53882599ms
time is 9696.76971436ms
time is 9708.24790001ms
time is 9767.36760139ms
time is 10067.9442883ms
Average time is 9677.78739333ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 16 14
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 11587.240696ms
time is 11979.0773392ms
time is 12009.3078613ms
time is 12288.7396812ms
time is 12287.9753113ms
time is 12342.8971767ms
time is 12366.2064075ms
time is 12517.8489685ms
time is 12530.4839611ms
time is 12551.4018536ms
time is 12571.3870525ms
time is 12745.9435463ms
time is 12774.9359608ms
time is 12772.8276253ms
Average time is 12380.448103ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 16 15
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 12026.0901451ms
time is 12155.3869247ms
time is 12754.327774ms
time is 12789.0229225ms
time is 12884.6731186ms
time is 13047.2779274ms
time is 13205.3160667ms
time is 13359.2040539ms
time is 13413.5987759ms
time is 13401.0264874ms
time is 13459.3603611ms
time is 13488.9039993ms
time is 13482.1488857ms
time is 13553.3621311ms
time is 13525.1400471ms
Average time is 13102.989308ms
alvaro@ryzen7:~/ruy$ python Benchmark.py stockfish uci 16 16
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
Running benchmark for stockfish
time is 13418.3039665ms
time is 13425.0230789ms
time is 13893.61763ms
time is 13932.5628281ms
time is 13970.7634449ms
time is 14104.1646004ms
time is 14165.6074524ms
time is 14167.3767567ms
time is 14235.7969284ms
time is 14338.8047218ms
time is 14351.2940407ms
time is 14347.7256298ms
time is 14389.1167641ms
time is 14402.2524357ms
time is 14448.8749504ms
time is 14461.3301754ms
Average time is 14128.2884628ms

[AndrewGrant]

For the newest runs posted

Code: Select all

    #Threads    Avg Time   LowerErr%   UpperErr%
           1      8467.0         0.0         0.0
           7      9451.0         0.3         0.5
           8      9678.0         2.7         4.0
          14     12380.0         6.4         3.2
          15     13103.0         8.2         3.4
          16     14128.0         5.0         2.4

This throws a wrench in things. Very low errors for 7thread, about the same as before for 14threads.

When I get my Threadripper up and running I'll run these tests a couple dozen times and average things out. I am trying here to convince not only you, but myself, that testing with hyper threads is alright. I'de quite like to (almost) double my testing speed.

[Leo]

Costco has a Ryzen 7 computer on sale for $600. (This is not a commercial, it just happens to be the lowest price I have seen.)

[Michael Sherwin]

Considering starting working on Hannibal again. My development computer is old and broken; any suggestions on what hardware I should purchase that gives me good bang for the buck for running lots of simultaneous test games? It would be a nice bonus if the machine was also good for online tournament play, but that is not a priority.

-Sam

Hi Sam, Long time. Anyway I have an I7 6 core.

If I analyze using Stockfish and Romi simultaneously and Stockfish is using:

5 threads Romi's node rate is 6950 kN/sec
6 threads Romi's node rate is 5896 kN/sec
8 threads Romi's node rate is 5264 kN/sec
10 threads Romi's node rate is 4861 kN/sec

[MikeB]

You can pick up a 16-core dual Xeon (2x Xeon e5-2670) on ebay for about $800. These are usually systems that were leased and are now being resold. The per-core performance is not stellar but is pretty decent. Most are sold with Windows but you can install Linux.

--Jon

Jon Dart has consistently provided the best value added information on buying a used or new computer setup. If you follow his advice, you can't go wrong with his value propositions,i.e., the best value for your money.