Different performance of equal executables

Discussion of chess software programming and technical issues.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
OliverBr
Posts: 669
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Different performance of equal executables

Post by OliverBr » Sat Oct 24, 2020 10:57 pm

Hello together,

investigating a performance worse than expected I fell over the following phenomena:

I have two Linux 64 executables named "olithink589" on a new Debian Linux with an idle AMD EPYC 7502P 32-Core Processor. One is in the current folded and another, older one is in "bin". The identity is checked by:

Code: Select all

diff ./olithink589 bin/olithink589
Now, I am running a performance check first on the underperforming executable "bin/olithink589":

Code: Select all

knps: 2090
Then testing the newer one "./olithink589":

Code: Select all

knps: 2221
This result is always reproducible and quite surprising: One executable is more than 6% faster than the other one. There are other executables in "bin" which are not affected by such slowdown.
Is there any explanation?
Chess Engine OliThink: http://brausch.org/home/chess
OliThink GitHub:https://github.com/olithink

abulmo2
Posts: 311
Joined: Fri Dec 16, 2016 10:04 am
Contact:

Re: Different performance of equal executables

Post by abulmo2 » Sun Oct 25, 2020 6:57 am

The environment of the executable, including the directory it runs from, its name, etc. can affect performance.
Videos about this subject:
https://www.youtube.com/watch?v=r-TLSBdHe1A
https://www.youtube.com/watch?v=koTf7u0v41o&t=1011s
and a paper:
https://www.researchgate.net/publicatio ... usly_Wrong
Richard Delorme

syzygy
Posts: 4995
Joined: Tue Feb 28, 2012 10:56 pm

Re: Different performance of equal executables

Post by syzygy » Sun Oct 25, 2020 12:26 pm

Type the following as root:
# echo 1 > /proc/sys/vm/drop_caches

Now try again.

RubiChess
Posts: 275
Joined: Fri Mar 30, 2018 5:20 am
Full name: Andreas Matthies

Re: Different performance of equal executables

Post by RubiChess » Sun Oct 25, 2020 12:40 pm

I noticed the same. Look here: viewtopic.php?f=7&t=73299 and here: https://github.com/AndyGrant/OpenBench/issues/50

I remember one case when syzygys drop_cache definitely helped so I'm using this before running tests on Linux and an "EmptyStandyList.exe" tool I found for Windows that should do similar.

Regards, Andreas

User avatar
flok
Posts: 258
Joined: Tue Jul 03, 2018 8:19 am
Full name: Folkert van Heusden
Contact:

Re: Different performance of equal executables

Post by flok » Sun Oct 25, 2020 1:35 pm

I would not use diff, but use e.g. md5sum.
Because I'm not sure what happens if the difference is a \cr or a 0x00.
OliverBr wrote:
Sat Oct 24, 2020 10:57 pm
Hello together,

investigating a performance worse than expected I fell over the following phenomena:

I have two Linux 64 executables named "olithink589" on a new Debian Linux with an idle AMD EPYC 7502P 32-Core Processor. One is in the current folded and another, older one is in "bin". The identity is checked by:

Code: Select all

diff ./olithink589 bin/olithink589
Now, I am running a performance check first on the underperforming executable "bin/olithink589":

Code: Select all

knps: 2090
Then testing the newer one "./olithink589":

Code: Select all

knps: 2221
This result is always reproducible and quite surprising: One executable is more than 6% faster than the other one. There are other executables in "bin" which are not affected by such slowdown.
Is there any explanation?
www.vanheusden.com: Micah / Embla / PuppetMaster / DeepBrutePos / Pos / Feeks

Sven
Posts: 3954
Joined: Thu May 15, 2008 7:57 pm
Location: Berlin, Germany
Full name: Sven Schüle
Contact:

Re: Different performance of equal executables

Post by Sven » Sun Oct 25, 2020 5:33 pm

"diff" would say "binary files differ" if they were different. But you don't need something like md5sum which requires to type several commands ... To determine whether two arbritrary files A and B are different at all, just type:

Code: Select all

cmp -s A B || echo "A and B are different"
If that produces empty output then A and B are binary identical. "cmp" should be slightly faster than "diff" because it does not try to present differences it finds but simply stops at the first difference.
Sven Schüle (engine author: Jumbo, KnockOut, Surprise)

User avatar
flok
Posts: 258
Joined: Tue Jul 03, 2018 8:19 am
Full name: Folkert van Heusden
Contact:

Re: Different performance of equal executables

Post by flok » Sun Oct 25, 2020 5:53 pm

Sven wrote:
Sun Oct 25, 2020 5:33 pm
"diff" would say "binary files differ" if they were different. But you don't need something like md5sum which requires to type several commands ... To determine whether two arbritrary files A and B are different at all, just type:

Code: Select all

cmp -s A B || echo "A and B are different"
If that produces empty output then A and B are binary identical. "cmp" should be slightly faster than "diff" because it does not try to present differences it finds but simply stops at the first difference.
several commands? for md5sum?
folkert@msi:~$ echo hallo > a
folkert@msi:~$ echo Hallo > b
folkert@msi:~$ md5sum a b
aee97cb3ad288ef0add6c6b5b5fae48a a
3290ec3c19a8a39362f7d70043f15627 b
1 command!
www.vanheusden.com: Micah / Embla / PuppetMaster / DeepBrutePos / Pos / Feeks

OliverBr
Posts: 669
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Re: Different performance of equal executables

Post by OliverBr » Sun Oct 25, 2020 7:34 pm

flok wrote:
Sun Oct 25, 2020 1:35 pm
I would not use diff, but use e.g. md5sum.
It doesn't matter.
Both files are absolutely identical. I also copied olithink589 from "bin" to another directory and got again the better (expected) performance of about 2220 knps. Only in the directory "bin" it's slower. Other executables in "bin" aren't slower.
Chess Engine OliThink: http://brausch.org/home/chess
OliThink GitHub:https://github.com/olithink

OliverBr
Posts: 669
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Re: Different performance of equal executables

Post by OliverBr » Sun Oct 25, 2020 7:44 pm

syzygy wrote:
Sun Oct 25, 2020 12:26 pm
Type the following as root:
# echo 1 > /proc/sys/vm/drop_caches
Yes, this solved it. Afterwards "bin/olithink589 performs exactly equal as "./olithink589" with 2220 knps.

Now this should explain the issue: What cache went probably wrong and how can this be avoided in the future?
Chess Engine OliThink: http://brausch.org/home/chess
OliThink GitHub:https://github.com/olithink

OliverBr
Posts: 669
Joined: Tue Dec 18, 2007 8:38 pm
Location: Munich, Germany
Full name: Dr. Oliver Brausch
Contact:

Re: Different performance of equal executables

Post by OliverBr » Sun Oct 25, 2020 7:51 pm

RubiChess wrote:
Sun Oct 25, 2020 12:40 pm
I noticed the same. Look here: viewtopic.php?f=7&t=73299 and here: https://github.com/AndyGrant/OpenBench/issues/50

I remember one case when syzygys drop_cache definitely helped so I'm using this before running tests on Linux and an "EmptyStandyList.exe" tool I found for Windows that should do similar.

Regards, Andreas
Yes, thank you, drop_cache helped!
PS: Up to 20 ELO drop is about the same I encountered in my case. I never would have thought that 6% performance loss yields to such a big difference in ELO.
Chess Engine OliThink: http://brausch.org/home/chess
OliThink GitHub:https://github.com/olithink

Post Reply