swami wrote:Thank you for the test. The design is interesting.
The rank order remains the same with more hash but we learn that Naum tends to plays better with more hash values. It's most affected by hash sizes relative to the rest of the engines.
Other Points to consider:
8 tests obviously wouldn't give clear picture. Though it could give rough strength of various engines.
More hash values will have effect only if the time control is more. For 10 seconds something less hash values works as it makes the search faster.
These tests are obviously strategical. So perhaps Stockfish is better at Tactics than Strategy than Naum.
The final result is consistent. The only thing though is I have been running a series of no book tournaments and Stockfish is out performing Naum and also Robbo* is out performing the rest using dual cores on a single core. More test suites are probably required to get more definite results as you say.