EGTB value for Rybka 3 again

[Jouni]

I had some spare time and played additional games (5 piece bases and 60/1' level). Results so far:

Rybka TB - Rybka NOTB 369,5 - 330,5 (+19)
Rybka TB - Stockfish 292 - 308 (-9)
Rybka NOTB - Stockfish 320,5 - 279,5 (+24)

What can be the explanation to 3. result, if not statistics? And how I calculate 95% error bar for n games?

Jouni

[ernest]

Rybka NOTB - Stockfish 320,5 - 279,5 (+24)? And how I calculate 95% error bar for n games?

Hi Jouni,

You did not give the draw rate.
If I assume it was 40% (of the 600 games), that means W+L=360 games
For close adversaries, the SD (standard deviation) formula simplifies to 0.5*sqrt(W+L) = 9.5
Twice that (=19) is the usual 95% error bar, so you get for Rybka NOTB: 320.5 ­­± 19, that is the range (301.5 - 339.5)

[Jouni]

Hi,

Draw rate was 49,9%.

Jouni

[Laskos]

Hi,

Draw rate was 49,9%.

Jouni

An exact formula for normal distribution (i.e. large number of games and not very skewed result) is for 1 standard deviation

+/- 100%*Sqrt(score*(1-score) - 0.25*NDraws/NGames)/Sqrt(NGames)

This is for 68% confidence. 95% confidence is 2 standard deviations, so double the previous result.

If you want in Elo points instead of percents just look at the Elo table in percentages or compute the Elo formula.

I ran a test with IvanHoes endgame 3-4-5 bitbases, and got 4 +/-3 Elo points advantage 95% confidence after 30,000 games. I guess Nalimovs give less or even harm, anyway you have to play several dozens of thousands of games. No joke.

Kai

[Jouni]

Probably testing with Rybka 3 wasn't the wisest idea, because there is TB access bug with MP search Because I don't have R4 I did short test
with 2.3.2 indicating +27 for EGTB access and also clear improvement against Stockfish 1.9! BTW do You really need always 1000s of games: in IPON list new engines rating after 200 games has never been much different than after 2000 games ?

Jouni

[ernest]

Draw rate was 49,9%.

If Draw rate was 49,9% (of the 600 games), that means W+L=300 games
For close adversaries, the SD (standard deviation) formula simplifies to 0.5*sqrt(W+L) = 8.66
Twice that (=17) is the usual 95% error bar, so you get for Rybka NOTB: 320.5 ­­± 17, that is the range (303.5 - 337.5)