In my blitz tests under MS XP, I have two binaries from the same source
which produce differing results. Diff says the sources are the same -- Diff says
the binaries differ. Both were compiled with MSVC++ 2005 express
edition.
More explicitly binary A has a 67% win rate +/- 1% each time I run the
gauntlet against a suite of opponents. The scores vs each opponent
differ from run to run but the overall score is consistant.
Binary B (same source just a recompile days later) scores between
36% and 47%. Much more variance and worse.
This made me scratch my head then I remembered Dann Corbitt
said that there can be issues with builds of C++ code due to
differences between intialization of something from one build to the
next.
Does anybody know what Dann was talking about and is there a
way to fix this behavior. I don't seem to get this behavior on Linux and
Dann did say such things would be compiler dependent.
Performance Variance due to C++
Moderators: hgm, Rebel, chrisw
-
- Posts: 2056
- Joined: Mon Mar 13, 2006 2:31 am
- Location: North Carolina, USA
-
- Posts: 3562
- Joined: Thu Mar 09, 2006 3:54 am
- Location: San Jose, California
Re: Performance Variance due to C++
I don't know squat about C or C++, but it has been my experience with some compilers that you are given a choice between either a very small execution file or one that runs really fast. You have the sources, how do they compare in size?
Bill
Bill
-
- Posts: 2056
- Joined: Mon Mar 13, 2006 2:31 am
- Location: North Carolina, USA
Re: Performance Variance due to C++
Same size.
Also, they produce the same results on benchmarks.
Also, they produce the same results on benchmarks.
-
- Posts: 12542
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Performance Variance due to C++
If you have not changed the compiler or the compiler settings or the source code at all, then your result is very unexpected.CRoberson wrote:In my blitz tests under MS XP, I have two binaries from the same source
which produce differing results. Diff says the sources are the same -- Diff says
the binaries differ. Both were compiled with MSVC++ 2005 express
edition.
More explicitly binary A has a 67% win rate +/- 1% each time I run the
gauntlet against a suite of opponents. The scores vs each opponent
differ from run to run but the overall score is consistant.
Binary B (same source just a recompile days later) scores between
36% and 47%. Much more variance and worse.
This made me scratch my head then I remembered Dann Corbitt
said that there can be issues with builds of C++ code due to
differences between intialization of something from one build to the
next.
Does anybody know what Dann was talking about and is there a
way to fix this behavior. I don't seem to get this behavior on Linux and
Dann did say such things would be compiler dependent.
If your engine learns, perhaps it is due to that effect.
-
- Posts: 1494
- Joined: Thu Mar 30, 2006 2:08 pm
Re: Performance Variance due to C++
One suggestion. Swap the order of the programs in whatever GUI you are using and see if the scores move with the specific program, or if it stay with the order of the programs. Maybe your GUI does something odd in terms of priorities fo search threads, or in how it healdes the engines based on who is first and second. For example, engine 1 might get all the hash memory it wants, but when setting up the process for engine 2, less has is left for it.? Just a guess.
Mark
Mark
-
- Posts: 224
- Joined: Mon Mar 12, 2007 7:31 pm
- Location: Bonn, Germany
Re: Performance Variance due to C++
There are many possiblities:CRoberson wrote: Diff says the sources are the same -- Diff says
the binaries differ. Both were compiled with MSVC++ 2005 express
edition.
- dependency on the order in which you load the engine into the gui, as mentioned by Mark
- vcproj-files differ (you only said that the sources do not)
- Visual studio sometimes has problems with a build (as oposed to a rebuild). What results do you get if you do a rebuild?
- Even if a build works, I'm not sure if it's supposed to produce identical binaries every time. If not, some bug like unitialized memory read of invalid pointer read might be fatal in one version but not the other. Consider to check your application with purify or a similar tool.
- The worse version was accidently built in debug mode.
- and lots more which currently don't come into my mind
-
- Posts: 2056
- Joined: Mon Mar 13, 2006 2:31 am
- Location: North Carolina, USA
Re: Performance Variance due to C++
Haven't looked at thatOnno Garms wrote: There are many possiblities:
- vcproj-files differ (you only said that the sources do not)
I always used rebuild.- Visual studio sometimes has problems with a build (as oposed to a rebuild). What results do you get if you do a rebuild?
Nope. Always a release mode version and I check the date and time- Even if a build works, I'm not sure if it's supposed to produce identical binaries every time. If not, some bug like unitialized memory read of invalid pointer read might be fatal in one version but not the other. Consider to check your application with purify or a similar tool.
- The worse version was accidently built in debug mode.
stamps on the binary to ensure that it is the one I just built.
-
- Posts: 1494
- Joined: Thu Mar 30, 2006 2:08 pm
Re: Performance Variance due to C++
I noticed the other day that if I load two copies of an identical program into the same GUI, the program sometiems take different amounts of memory. I am not sure why...I do not have them set to different hash table sizes. Maybe there is not enough physical memory left and one process gets less memory or something. If I figure out why I will let you know, but different amounts of memory could explain some search differences.