Opening testing suites efficiency
Posted: Wed Jun 21, 2017 2:34 pm
I wanted to test (with Komodo) the sensitivity to doubling time control of different known to me opening suites. The time control was 6''+0.06'' versus 3''+0.03''. One overlooked fact, with for example Stockfish Framework standard opening suite 2 moves_v1.epd, is that those 2 random moves distort all the opening phase of the play as shown in this thread:
http://www.talkchess.com/forum/viewtopi ... =0&t=63763
I separated the suites in ones distorting the openings of Standard Chess (and they have higher sensitivity here), and those playing reasonable openings. The important thing to look here is the t-value (sensitivity) of the difference, and the better indicator is "Normalized ELO" t-value, and not "ELO" t-value.
"Normalized ELO" is proposed recently by Michel Van den Bergh here:
http://talkchess.com/forum/viewtopic.ph ... t&start=20
Here are the results:
Opening suites which distort the opening phase
Opening suites with reasonable openings
The shortcomings of the two top suites: Chess960 - only 960 starting positions. 3moves_GM - only 1170 positions. In fast games that shouldn't be a problem even with tens of thousands of games, no game will be a repeat.
http://www.talkchess.com/forum/viewtopi ... =0&t=63763
I separated the suites in ones distorting the openings of Standard Chess (and they have higher sensitivity here), and those playing reasonable openings. The important thing to look here is the t-value (sensitivity) of the difference, and the better indicator is "Normalized ELO" t-value, and not "ELO" t-value.
"Normalized ELO" is proposed recently by Michel Van den Bergh here:
http://talkchess.com/forum/viewtopic.ph ... t&start=20
Here are the results:
Opening suites which distort the opening phase
Code: Select all
Chess960.epd
Score of K2 vs K1: 1172 - 119 - 709 [0.763] 2000
ELO difference: 203.35 +/- 12.78
t-value = 15.9
Normalized ELO difference: 0.868
t-value = 19.8
2moves_v1.epd
Score of K2 vs K1: 1164 - 144 - 692 [0.755] 2000
ELO difference: 195.51 +/- 12.91
t-value = 15.1
Normalized ELO difference: 0.813
t-value = 18.6
Code: Select all
3moves_GM.epd
Score of K2 vs K1: 1085 - 126 - 789 [0.740] 2000
ELO difference: 181.48 +/- 12.11
t-value = 15.0
Normalized ELO difference: 0.782
t-value = 17.8
8moves_v3.pgn
Score of K2 vs K1: 1013 - 113 - 874 [0.725] 2000
ELO difference: 168.40 +/- 11.45
t-value = 14.7
Normalized ELO difference: 0.749
t-value = 17.1
8moves_GM.pgn
Score of K2 vs K1: 1002 - 137 - 861 [0.716] 2000
ELO difference: 160.85 +/- 11.57
t-value = 13.9
Normalized ELO difference: 0.699
t-value = 15.9