TalkChess.com

[Robert Hyatt]

Here are 32 80-game matches, with +=a win, -=a loss and = means drawn. The 80 game matches are between the two same opponents, same 40 starting opening positions one with white one with black, same time control, same everything. Look at this and then tell me that two programs play the same games often if nothing is changed...

What is interesting is to look at each column (not so easy as formatted by HTML). Each column represents the same starting position and color. You might expect for each column to have the same character/result since the two programs, the time control, the starting position and starting color are identical within a column. But expectations and actual results don't match up. Some columns are pretty deterministic, although finding any column with identical results from top to bottom is rare. Then compare any two rows to see how often a match produces the same or very close to the same results game by game.

Variance is for real, and it is a huge problem.