Benchmarking in .Net

[chessguy]

Writing a new engine in c#, and I'm having trouble getting reliable benchmarking results. I've tried both wrapping my perfts in nunit tests, and also just benchmarking the binary by hacking in special command line commands. But it seems like my results vary somewhat wildly.

Any other .Net developers out there? How do you do it?

[jesper_nielsen]

Writing a new engine in c#, and I'm having trouble getting reliable benchmarking results. I've tried both wrapping my perfts in nunit tests, and also just benchmarking the binary by hacking in special command line commands. But it seems like my results vary somewhat wildly.

Any other .Net developers out there? How do you do it?

I have both perft and something I call perftEval. PerftEval is the same as perft, but at the leafs, the evaluation function is called.

I don't really use them as benchmarks though.

When testing performance improvements, i usually just run a search from the startposition for a fixed number of seconds. Typically a minute or so. Sometimes as long as 10 minutes. And then compare the number of nodes searched before and after the change.

This is ONLY for pure performance changes. Not for anything that will change the shape of the search tree.

When I find a fast and reliable way to test changes to the search, I will become very famous! And rich!

Edit: Assuming you need the benchmark data to compare changes locally, not to compare the same version in different environments!

Kind regards,
Jesper

[Dann Corbit]

Writing a new engine in c#, and I'm having trouble getting reliable benchmarking results. I've tried both wrapping my perfts in nunit tests, and also just benchmarking the binary by hacking in special command line commands. But it seems like my results vary somewhat wildly.

Any other .Net developers out there? How do you do it?

Are you using the built in profiler?

Suggestion:
Benchmark for at least 20 minutes. Quick bencharks will be largely noise from startup overhead, etc.