Mostly a theoretical question.
An Oracle opening book is a book that contains NO more than 100 openings.
Basically, you test two engine with this book.
Based on the score results of the 200 games...it will always determine with 99% accuracy which engine is superior...or if ELO difference is negligible.
So instead of testing 100,000s of games...one only needs to run 200 games.
............
My first thought of an Oracle book is to use the first 20 opening moves. (Perft 1)
Would an Oracle Test-book be possible?
Moderators: hgm, Rebel, chrisw
-
- Posts: 27811
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Would an Oracle Test-book be possible?
It depends on what you consider 'negligible'.
With 200 games you will have a 40%/sqrt(200) = 2.8% standard deviation, or about 20 Elo. To have 99% confidence you need about 2.5 STD, or 50 Elo. As 50 Elo is indeed a negligible Elo difference, I guess most sets of 100 openings that are sufficiently different will do fine.
With 200 games you will have a 40%/sqrt(200) = 2.8% standard deviation, or about 20 Elo. To have 99% confidence you need about 2.5 STD, or 50 Elo. As 50 Elo is indeed a negligible Elo difference, I guess most sets of 100 openings that are sufficiently different will do fine.
-
- Posts: 154
- Joined: Tue May 17, 2011 8:12 pm
Re: Would an Oracle Test-book be possible?
What I am trying to say is that SF Test framework plays tens of thousands of games to determine if a patch should be approved or not.
Instead of playing all those games...the Oracle book can make the same decision by only playing 200 games.
What I mean by negligible is ELO between the two engines are +/- 1-2 ELO.
Instead of playing all those games...the Oracle book can make the same decision by only playing 200 games.
What I mean by negligible is ELO between the two engines are +/- 1-2 ELO.
-
- Posts: 27811
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Would an Oracle Test-book be possible?
Well, for 1 Elo with 99% confidence, you will need 200,000 games. If the book is perfect (i.e. if the initial positions are all sufficiently different to guarantee independent games). This cannot be helped.
-
- Posts: 558
- Joined: Sat Mar 25, 2006 8:27 pm
Re: Would an Oracle Test-book be possible?
I think the other piece of the equation is the game score.voyagerOne wrote:What I am trying to say is that SF Test framework plays tens of thousands of games to determine if a patch should be approved or not.
Instead of playing all those games...the Oracle book can make the same decision by only playing 200 games.
What I mean by negligible is ELO between the two engines are +/- 1-2 ELO.
With 200 games, you can determine that A is better than B, but it has to score 55% to do so. With 1000 games, you prove the same thing with a 52% score, but that is actually a lower hurdle because the random variation is lower.
The fewer games you test with, the more spectacularly you have to dominate the opponent in order to prove actual superiority.
-
- Posts: 154
- Joined: Tue May 17, 2011 8:12 pm
Re: Would an Oracle Test-book be possible?
The key word here is "Oracle".
-
- Posts: 27811
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Would an Oracle Test-book be possible?
You cannot make an oracle from opening lines anymore than you can make a nuclear reactor by stacking ice cubes. And if you had a true oracle, you would need no lines at all. You would just feed it the names, and ask which was better.