There are two main reasons:JP wrote:I don't get why you turn LMR off during move ordering tests.
With LMR on, it is not possible to determine whether move ordering scheme A is better than move ordering scheme B simply by comparing node counts for a big number of positions, because the move ordering affects the average number of reduced moves per node. It is possible that move ordering scheme A will consistently give smaller trees than move ordering scheme B simply because scheme A results in a bigger number of moves being reduced. Scheme A will be a bit faster, while scheme B will be a bit more precise. Which of the two performs better in actual play can only be determined by playing a huge number of test games.
The most obvious example of this phenomenon is the ordering of losing captures. I, like most other programmers, never reduce captures, regardless of their position in the move list. If losing captures are searched early (for instance between the equal captures and the killer moves), the average number of reduced moves per position will go down compared to when they are searched late (at the end of the move list, for example).
The second problem is that LMR warps the shape of the tree in essentially random ways. A tiny change in move ordering will often change the the score, the PV, or even the best move for a given position at a search to depth N. The number of nodes may easily differ by a factor of 2 or more. Of course this sort of thing occurs without LMR as well, but not nearly as often. The variance for all sorts of statistics is much bigger with LMR on, which means that you need a much bigger set of tests before you can make any statistically valid conclusions.
Tord