Performance Variance due to C++

[CRoberson]

In my blitz tests under MS XP, I have two binaries from the same source
which produce differing results. Diff says the sources are the same -- Diff says
the binaries differ. Both were compiled with MSVC++ 2005 express
edition.

More explicitly binary A has a 67% win rate +/- 1% each time I run the
gauntlet against a suite of opponents. The scores vs each opponent
differ from run to run but the overall score is consistant.

Binary B (same source just a recompile days later) scores between
36% and 47%. Much more variance and worse.

This made me scratch my head then I remembered Dann Corbitt
said that there can be issues with builds of C++ code due to
differences between intialization of something from one build to the
next.

Does anybody know what Dann was talking about and is there a
way to fix this behavior. I don't seem to get this behavior on Linux and
Dann did say such things would be compiler dependent.

[Bill Rogers]

I don't know squat about C or C++, but it has been my experience with some compilers that you are given a choice between either a very small execution file or one that runs really fast. You have the sources, how do they compare in size?
Bill

[CRoberson]

Same size.

Also, they produce the same results on benchmarks.

[Dann Corbit]

In my blitz tests under MS XP, I have two binaries from the same source
which produce differing results. Diff says the sources are the same -- Diff says
the binaries differ. Both were compiled with MSVC++ 2005 express
edition.

More explicitly binary A has a 67% win rate +/- 1% each time I run the
gauntlet against a suite of opponents. The scores vs each opponent
differ from run to run but the overall score is consistant.

Binary B (same source just a recompile days later) scores between
36% and 47%. Much more variance and worse.

This made me scratch my head then I remembered Dann Corbitt
said that there can be issues with builds of C++ code due to
differences between intialization of something from one build to the
next.

Does anybody know what Dann was talking about and is there a
way to fix this behavior. I don't seem to get this behavior on Linux and
Dann did say such things would be compiler dependent.

If you have not changed the compiler or the compiler settings or the source code at all, then your result is very unexpected.

If your engine learns, perhaps it is due to that effect.

[mjlef]

One suggestion. Swap the order of the programs in whatever GUI you are using and see if the scores move with the specific program, or if it stay with the order of the programs. Maybe your GUI does something odd in terms of priorities fo search threads, or in how it healdes the engines based on who is first and second. For example, engine 1 might get all the hash memory it wants, but when setting up the process for engine 2, less has is left for it.? Just a guess.

Mark

[Onno Garms]

Diff says the sources are the same -- Diff says
the binaries differ. Both were compiled with MSVC++ 2005 express
edition.

There are many possiblities:
- dependency on the order in which you load the engine into the gui, as mentioned by Mark
- vcproj-files differ (you only said that the sources do not)
- Visual studio sometimes has problems with a build (as oposed to a rebuild). What results do you get if you do a rebuild?
- Even if a build works, I'm not sure if it's supposed to produce identical binaries every time. If not, some bug like unitialized memory read of invalid pointer read might be fatal in one version but not the other. Consider to check your application with purify or a similar tool.
- The worse version was accidently built in debug mode.
- and lots more which currently don't come into my mind

[CRoberson]

There are many possiblities:
- vcproj-files differ (you only said that the sources do not)

Haven't looked at that

- Visual studio sometimes has problems with a build (as oposed to a rebuild). What results do you get if you do a rebuild?

I always used rebuild.

- Even if a build works, I'm not sure if it's supposed to produce identical binaries every time. If not, some bug like unitialized memory read of invalid pointer read might be fatal in one version but not the other. Consider to check your application with purify or a similar tool.
- The worse version was accidently built in debug mode.

Nope. Always a release mode version and I check the date and time
stamps on the binary to ensure that it is the one I just built.

[mjlef]

I noticed the other day that if I load two copies of an identical program into the same GUI, the program sometiems take different amounts of memory. I am not sure why...I do not have them set to different hash table sizes. Maybe there is not enough physical memory left and one process gets less memory or something. If I figure out why I will let you know, but different amounts of memory could explain some search differences.