Make engine stop repeating moves in a clearly won position

[Ras]

in several 20 game matches.

Please keep in mind that for a 95% confidence interval among about roughly equal opponents, your error margin is +/- sqrt(N). 20 games matches don't tell much.

[mvanthoor]

in several 20 game matches.

Please keep in mind that for a 95% confidence interval among about roughly equal opponents, your error margin is +/- sqrt(N). 20 games matches don't tell much.

Everybody keeps saying that. I know. It's just a very rough estimate. This week I'm going to build an opening book, and start my own rating pool with my engine + 10 others. I'll have them all play 20 games against each other engine, which will mean each engine will play 200 games, just like in CCRL. The total will be 2200 games. The engines will be between 1500 and 1800 Elo. Then I will have my own mini-rating list and and I'll know where this first version of Rustic sits (approximately) on the CCRL blitz list. After that I can start adding features and have it run gauntlets and one-to-one self-play matches.

[Ras]

Everybody keeps saying that. I know. It's just a very rough estimate.

Well, interpreting 52% in 50 games as +14 Elo is questionable because the error margin in 50 games is already +/- 100 Elo. In order to have 95% confidence that 52% represents any real improvement at all (still without quantification how much), you'd need 2500 games.

The total will be 2200 games.

That looks better and will give an Elo error margin of about +/- 50 Elo for each engine. That should be fine for a ballpark estimation of Rustic.