inconsistent performance

[Don]

I'm having a serious problem with performance benchmarks with my chess program. I cannot get a consistent set of times from one run to the next. The difference isn't minor, it's major. For instance one position went from 167.3 seconds to 104.5 to do a 15 ply search. Another when from 43.7 to 27.1. I can run this many times and get a different result each time, some problems running faster and some running slower and there seems to be no pattern to it.

I have the ability to test this by CPU time but it makes no difference, same basic behavior. I run these on an unloaded core 2 duo machine in Linux and top, ps or uptime shows the load to be very close to zero when I start the test.

The program is deterministic, so I get the same score, number of nodes searched, and principal variation - identical in every way. It's really making life difficult for performance benchmarking!

Has anyone else run into this? I don't remember this being a problem before. I strongly suspect some kind of caching issue. As far as I can see, however, the program is deterministic in every way when I want it to be.

- Don

[Gerd Isenberg]

I'm having a serious problem with performance benchmarks with my chess program. I cannot get a consistent set of times from one run to the next. The difference isn't minor, it's major. For instance one position went from 167.3 seconds to 104.5 to do a 15 ply search. Another when from 43.7 to 27.1. I can run this many times and get a different result each time, some problems running faster and some running slower and there seems to be no pattern to it.

I have the ability to test this by CPU time but it makes no difference, same basic behavior. I run these on an unloaded core 2 duo machine in Linux and top, ps or uptime shows the load to be very close to zero when I start the test.

The program is deterministic, so I get the same score, number of nodes searched, and principal variation - identical in every way. It's really making life difficult for performance benchmarking!

Has anyone else run into this? I don't remember this being a problem before. I strongly suspect some kind of caching issue. As far as I can see, however, the program is deterministic in every way when I want it to be.

- Don

I can't remember such extreme spreading. But I think Bob once told that someone at a tournament started his program x-times, until it run some test with sufficient nps

Have you tried other compilers (settings) or hardware? Or other chess programs on the same hardware - if it behaves similar, something might be broken with your hardware or configuration. May be your program or special build is close to some limits and therefor let say sensible where the code and data becomes physically located in memory (some none cached areas?!). If your computer has multiple cores, process affinity might be an issue. Can you trace number page faults similar to windows task-manager?

Gerd

[Don]

I'm having a serious problem with performance benchmarks with my chess program. I cannot get a consistent set of times from one run to the next. The difference isn't minor, it's major. For instance one position went from 167.3 seconds to 104.5 to do a 15 ply search. Another when from 43.7 to 27.1. I can run this many times and get a different result each time, some problems running faster and some running slower and there seems to be no pattern to it.

I have the ability to test this by CPU time but it makes no difference, same basic behavior. I run these on an unloaded core 2 duo machine in Linux and top, ps or uptime shows the load to be very close to zero when I start the test.

The program is deterministic, so I get the same score, number of nodes searched, and principal variation - identical in every way. It's really making life difficult for performance benchmarking!

Has anyone else run into this? I don't remember this being a problem before. I strongly suspect some kind of caching issue. As far as I can see, however, the program is deterministic in every way when I want it to be.

- Don

I can't remember such extreme spreading. But I think Bob once told that someone at a tournament started his program x-times, until it run some test with sufficient nps

Have you tried other compilers (settings) or hardware? Or other chess programs on the same hardware - if it behaves similar, something might be broken with your hardware or configuration. May be your program or special build is close to some limits and therefor let say sensible where the code and data becomes physically located in memory (some none cached areas?!). If your computer has multiple cores, process affinity might be an issue. Can you trace number page faults similar to windows task-manager?

Gerd

I said there was no pattern, but there seems to be a partial pattern. The 20 problem set seems to run at 2 basic speeds which I will call fast or slow. Because I can get a run where most every position is running 20 or 30 percent slower than when it is behaving well. But occasionally in 2 sets that are timing about the same 1 problem will be be substantially slower or faster.

I've never seen anything quite this odd.

[Don]

I'm having a serious problem with performance benchmarks with my chess program. I cannot get a consistent set of times from one run to the next. The difference isn't minor, it's major. For instance one position went from 167.3 seconds to 104.5 to do a 15 ply search. Another when from 43.7 to 27.1. I can run this many times and get a different result each time, some problems running faster and some running slower and there seems to be no pattern to it.

I have the ability to test this by CPU time but it makes no difference, same basic behavior. I run these on an unloaded core 2 duo machine in Linux and top, ps or uptime shows the load to be very close to zero when I start the test.

The program is deterministic, so I get the same score, number of nodes searched, and principal variation - identical in every way. It's really making life difficult for performance benchmarking!

Has anyone else run into this? I don't remember this being a problem before. I strongly suspect some kind of caching issue. As far as I can see, however, the program is deterministic in every way when I want it to be.

- Don

I can't remember such extreme spreading. But I think Bob once told that someone at a tournament started his program x-times, until it run some test with sufficient nps

Have you tried other compilers (settings) or hardware? Or other chess programs on the same hardware - if it behaves similar, something might be broken with your hardware or configuration. May be your program or special build is close to some limits and therefor let say sensible where the code and data becomes physically located in memory (some none cached areas?!). If your computer has multiple cores, process affinity might be an issue. Can you trace number page faults similar to windows task-manager?

Gerd

I said there was no pattern, but there seems to be a partial pattern. The 20 problem set seems to run at 2 basic speeds which I will call fast or slow. Because I can get a run where most every position is running 20 or 30 percent slower than when it is behaving well. But occasionally in 2 sets that are timing about the same 1 problem will be be substantially slower or faster.

I've never seen anything quite this odd.

I think the problem is processor affinity (or lack of it!) For all 20 positions I get average times rangin from 7.26 to 18.26 on a machine that is not loaded! Same total number of nodes.

The best time is 7.26 and it was run using taskset, which can force a program to use only a specified processor(s) And as it turns out I get consistent times (and the best times) only when I use taskset for this. I'm not sure what would happen if the machine was loaded with other tasks however and I'm not sure why this should make a difference but I assume it is memory and cache related.


drd@greencheeks:~/Games/Chess/ChessPrograms/mychess$ for f in n?.log ; do grep ' 13 ' $f | ave 1 | tail -1 ; done
18.2600 AVERAGE of 20 entries
12.8500 AVERAGE of 20 entries
9.5700 AVERAGE of 20 entries
9.3000 AVERAGE of 20 entries
11.3800 AVERAGE of 20 entries
7.3250 AVERAGE of 20 entries
10.5700 AVERAGE of 20 entries
7.2650 AVERAGE of 20 entries

[kranium]

I'm having a serious problem with performance benchmarks with my chess program. I cannot get a consistent set of times from one run to the next. The difference isn't minor, it's major. For instance one position went from 167.3 seconds to 104.5 to do a 15 ply search. Another when from 43.7 to 27.1. I can run this many times and get a different result each time, some problems running faster and some running slower and there seems to be no pattern to it.

I have the ability to test this by CPU time but it makes no difference, same basic behavior. I run these on an unloaded core 2 duo machine in Linux and top, ps or uptime shows the load to be very close to zero when I start the test.

The program is deterministic, so I get the same score, number of nodes searched, and principal variation - identical in every way. It's really making life difficult for performance benchmarking!

Has anyone else run into this? I don't remember this being a problem before. I strongly suspect some kind of caching issue. As far as I can see, however, the program is deterministic in every way when I want it to be.

- Don

I can't remember such extreme spreading. But I think Bob once told that someone at a tournament started his program x-times, until it run some test with sufficient nps

Have you tried other compilers (settings) or hardware? Or other chess programs on the same hardware - if it behaves similar, something might be broken with your hardware or configuration. May be your program or special build is close to some limits and therefor let say sensible where the code and data becomes physically located in memory (some none cached areas?!). If your computer has multiple cores, process affinity might be an issue. Can you trace number page faults similar to windows task-manager?

Gerd

I said there was no pattern, but there seems to be a partial pattern. The 20 problem set seems to run at 2 basic speeds which I will call fast or slow. Because I can get a run where most every position is running 20 or 30 percent slower than when it is behaving well. But occasionally in 2 sets that are timing about the same 1 problem will be be substantially slower or faster.

I've never seen anything quite this odd.

I think the problem is processor affinity (or lack of it!) For all 20 positions I get average times rangin from 7.26 to 18.26 on a machine that is not loaded! Same total number of nodes.

The best time is 7.26 and it was run using taskset, which can force a program to use only a specified processor(s) And as it turns out I get consistent times (and the best times) only when I use taskset for this. I'm not sure what would happen if the machine was loaded with other tasks however and I'm not sure why this should make a difference but I assume it is memory and cache related.


drd@greencheeks:~/Games/Chess/ChessPrograms/mychess$ for f in n?.log ; do grep ' 13 ' $f | ave 1 | tail -1 ; done
18.2600 AVERAGE of 20 entries
12.8500 AVERAGE of 20 entries
9.5700 AVERAGE of 20 entries
9.3000 AVERAGE of 20 entries
11.3800 AVERAGE of 20 entries
7.3250 AVERAGE of 20 entries
10.5700 AVERAGE of 20 entries
7.2650 AVERAGE of 20 entries

in order to identify whether the issue is system related or program related, have you tried running the test(s) immediately after a system reboot...(or on an alternate system)?

i.e is the program releasing memory/killing processes/stopping threads/exiting, etc. properly?
i.e. are you sure you're starting clean each time?

Norm

[bob]

I'm having a serious problem with performance benchmarks with my chess program. I cannot get a consistent set of times from one run to the next. The difference isn't minor, it's major. For instance one position went from 167.3 seconds to 104.5 to do a 15 ply search. Another when from 43.7 to 27.1. I can run this many times and get a different result each time, some problems running faster and some running slower and there seems to be no pattern to it.

I have the ability to test this by CPU time but it makes no difference, same basic behavior. I run these on an unloaded core 2 duo machine in Linux and top, ps or uptime shows the load to be very close to zero when I start the test.

The program is deterministic, so I get the same score, number of nodes searched, and principal variation - identical in every way. It's really making life difficult for performance benchmarking!

Has anyone else run into this? I don't remember this being a problem before. I strongly suspect some kind of caching issue. As far as I can see, however, the program is deterministic in every way when I want it to be.

- Don

I can usually see a 1% max deviation in time, caused by a program being loaded into a different set of real pages each time it is run, which can affect memory-to-cache mapping. But I've never seen anything like that with one exception...

You didn't mention hardware. Is it possible you have two different kinds of memory? I had one of the original Toshiba pentium laptops, with 8 megs of ram. Which ran on the usual 64 bit bus. But I added another 16mb of ram, and it turns out their expansion slot was 32 bits and not 64 bits. Which meant I could run once, get lucky, and get loaded into the first 8mb and run like the blazes. The next time, I could get loaded into the upper 16mb and run dog slow. And other times I would get a combination with the speed varying all over the place.

I can run the same position N times on my laptop and see almost zero variance using linux...

[Aleks Peshkov]

I have little experience with Linux, but on Windows system power management is common reason. It is possible to observe that CPU runs sometimes on designed speed, sometimes on half frequency clock. Overheating can also make Intel CPU to drop speed without any software control.

[hgm]

It will be a bit difficult to make sensible comments on this if you don't tell us what hardware you are running on, and what your engine tries to use of this (e.g. is it SMP, how many cores, what hash setting, does it use huge tables).

[Bill Rogers]

Don
I have experienced almost the same thing. I was timming a certain move in a chess game and thought the result was fine until I did it a second time. It was then that I noticed that even though I was not using my modem the computer was keeping it activated so I turn it off. To my supprise my timing improved drastically. So it appears that some things running in the background can effect your timing on many things.
Bill

[Don]

It will be a bit difficult to make sensible comments on this if you don't tell us what hardware you are running on, and what your engine tries to use of this (e.g. is it SMP, how many cores, what hash setting, does it use huge tables).

I'm running on a core 2 duo and the program is not multi-threaded. The hash tables are set to 32M.

I also use a large data structure for evaluation which are similar to pawn structure hash tables but do much more. But the problem persists even when I set these tables to a very small size.

I do use a lot of tables, but I think think this is out NOT of the ordinary for typical 64 bit programs.