Variable results in benchmarking making changes difficult to measure

Discussion of chess software programming and technical issues.

Moderators: hgm, chrisw, Rebel

ydebilloez
Posts: 171
Joined: Tue Jun 27, 2017 11:01 pm
Location: Lubumbashi
Full name: Yves De Billoëz

Variable results in benchmarking making changes difficult to measure

Post by ydebilloez »

Dear all,

When running benchmarks, I see variable results on mac and linux in between each run. Can someone hint me to fix the execution speed in following conditions: system idle and system busy. I am thinking of real time linux or fixing processor speed. I can bear with short differences in runtime by false clock measurements in sub-second runtimes but it would be nice to fix this as well.

Code: Select all

$./comparebench.sh 
~/chess/belofte/dist/Linux-x86-64 ~/chess/belofte/test
...
bench Brute Force Time: 57.308 Nodes: 8470712 Total N.: 8870156 NPS: 154778
...
$ ./comparebench.sh 
...
bench Brute Force Time: 58.077 Nodes: 8470712 Total N.: 8870156 NPS: 152729
Subsequent runs are sometimes more than 1 second off.
Thanks,
Yves
Yves De Billoëz @ macchess belofte chess
Once owner of a Mephisto I, II, challenger, ... chess computer.
Ras
Posts: 2604
Joined: Tue Aug 30, 2016 8:19 pm
Full name: Rasmus Althoff

Re: Variable results in benchmarking making changes difficult to measure

Post by Ras »

ydebilloez wrote: Fri Oct 25, 2024 12:24 pmI am thinking of real time linux or fixing processor speed.
I disable the CPU boost during testing so that the CPU will stay at the base clock.

Disable boost:

Code: Select all

echo "0" | sudo tee /sys/devices/system/cpu/cpufreq/boost
Enable boost:

Code: Select all

echo "1" | sudo tee /sys/devices/system/cpu/cpufreq/boost
Check boost status:

Code: Select all

cat /sys/devices/system/cpu/cpufreq/boost
If you're interested, I've made a shell script and a Zenity script around that so that I can easily access that from the desktop start menu.
Rasmus Althoff
https://www.ct800.net
ydebilloez
Posts: 171
Joined: Tue Jun 27, 2017 11:01 pm
Location: Lubumbashi
Full name: Yves De Billoëz

Re: Variable results in benchmarking making changes difficult to measure

Post by ydebilloez »

Ras wrote: Fri Oct 25, 2024 3:39 pm I disable the CPU boost during testing so that the CPU will stay at the base clock.

Disable boost:

Code: Select all

echo "0" | sudo tee /sys/devices/system/cpu/cpufreq/boost
Enable boost:

Code: Select all

echo "1" | sudo tee /sys/devices/system/cpu/cpufreq/boost
Hi Ras,

Thanks for your suggestion. On my pc, I get Permission denied when running this. But you gave me a clue. This is the solution I found working to some extend.

Code: Select all

# echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
And setting it back to "0" in to resume to normal. Turbo enabled, I get variations of 2-5%. With the turbo disabled, I get variations well below 0.5% on long runs (at the expense of 20% execution speed), both on AC and battery. So perfectly usable.

Thanks for your feedback,
Yves
Yves De Billoëz @ macchess belofte chess
Once owner of a Mephisto I, II, challenger, ... chess computer.