What is strongest Rybka 4? -- and is Houdini cheating ?

[Don]

I know this an old thread, but cheating like this is a pretty serious allegation.

So I checked it out myself. I see no evidence whatsoever that Houdini cheats like this.

I basically set up a test where I sample resource usage over a period of time while playing a game between Stockfish and Houdini and I get very close to 50/50 ratio.

Now I would like to see the evidence that Houdini cheats and what made someone say that.

I'm getting 50.78 CPU utilization for Houdini and 48.91 and I assume the tester and other processes are using some resources. The small difference is well under the margin of statistically credulity.

Robert says that Houdini uses exactly the number to threads you set it to. That means that he does not do I/O in a separate thread. I don't know how this is handled in windows but in unix operating system you can poll for i/o (which will consume CPU resources even when not thinking) or it can be done other ways with notifications from the operating system.

If you do the polling manually in a tight loop there must be a way to yield the thread or else you are consuming CPU resources as if you are thinking. This can be done via usleep() in linux. It is possible that too short a delay can also consume a non-trivial amount of resources.

One thing to keep in mind is that there is not a 100% fair way to test 2 programs on the same computer. You cannot live in the same neighborhood without having at least SOME impact on your neighbors. Same with computer resources. To be as fair as possible you should run each program on it's own dedicated machine with a third machine arbiter. But for most of us this is a completely impractical setup, especially for developers who depend on massive automated testing.

Don

If testing was done on two separate computers and still maintained the same elo difference between Houdini and Rybka then that would suggest there is no cheating going on.

There is no cheating going on.
Even on a single computer you can easily establish the following via the Task Manager by displaying the Threads and Total CPU time columns:

1) Unlike most other programs, Houdini always uses the exact number of threads that is requested. If you have specified Threads=4, you'll find exactly 4 threads running in the Task Manager. There are no idle threads that could steal CPU from the opponent.

2) The CPU total time in the Task Manager will correspond perfectly to the time that has elapsed on the GUI clock. If the program has used 20 seconds in the GUI, and 4 threads are running, the total CPU time will be very close to 80 seconds.

3) Very often, the total CPU time of other engines will be higher than Houdini's. As far as I know, Houdini is more conservative in its time usage than for example Stockfish or Ivanhoe.

Robert