Critter 1.4a x64 SSE4 vs Houdini 2.0c Pro x64 match results

[Houdini]

My actual source is a statistical textbook, but the Wikipedia entry for "prediction interval" also has the formula for computing the endpoints of the interval. For computing the variance, I used the approximation Error = 100%* Sqrt((score*(1-score)-0.25*draw%)/games).

You cannot use this variance for predicting a second engine match from a first match engine result, in that case you need to multiply it by 2.
See the "prediction interval" Wikepedia page you refer to, paragraph "Unknown mean, known variance".
http://en.wikipedia.org/wiki/Prediction_interval

The variance for predicting a future "observation" is (1 + 1/n) larger than the known population variance, with n the number of previous observations.
In this case we only have a single previous "observation" (engine match), so n = 1 and the variance is twice as high as the population variance.
This leads to 1.4 times higher error bars than the formula you cite above, exactly like I said in my previous post.

Robert

[Adam Hair]

My actual source is a statistical textbook, but the Wikipedia entry for "prediction interval" also has the formula for computing the endpoints of the interval. For computing the variance, I used the approximation Error = 100%* Sqrt((score*(1-score)-0.25*draw%)/games).

You cannot use this variance for predicting a second engine match from a first match engine result, in that case you need to multiply it by 2.
See the "prediction interval" Wikepedia page you refer to, paragraph "Unknown mean, known variance".
http://en.wikipedia.org/wiki/Prediction_interval

The variance for predicting a future "observation" is (1 + 1/n) larger than the known population variance, with n the number of previous observations.
In this case we only have a single previous "observation" (engine match), so n = 1 and the variance is twice as high as the population variance.
This leads to 1.4 times higher error bars than the formula you cite above, exactly like I said in my previous post.

Robert

You are correct. I screwed up by making n = 150, the number of games in Ingo's sample, instead of 1, the number of samples upon which the prediction was being made.

Thanks for setting me straight.

[Houdini]

Hi, Robert! I see you are using term " variability and error margins in chess engine testing" Question for You: If I make test with reversible starting positions ( and I always do that ); are those " variability and error margins in chess engine testing" the same if I dont use reversible starting positions?


Regards!

I assume that with "reversible starting positions" you mean that the engines play each starting position twice, with white and black.

It's a complex matter. With reversed openings you increase the accuracy of the result, but you also increase the variance of the result.
The "accuracy" is increased because any bias for white or black is eliminated. By playing more and more games you will ultimately converge to the "correct" result.
The "variance" is larger because the 2 games from the same starting position will not be completely independent. This means that you'll need to play more games to reduce the uncertainty margins.

Robert

[marijan]

I assume that with "reversible starting positions" you mean that the engines play each starting position twice, with white and black.

Yup!

It's a complex matter...

I know that... Thanks for the answer...


Regards...

[beram]

Hi, Robert! I see you are using term " variability and error margins in chess engine testing" Question for You: If I make test with reversible starting positions ( and I always do that ); are those " variability and error margins in chess engine testing" the same if I dont use reversible starting positions?


Regards!

It's a complex matter. With reversed openings you increase the accuracy of the result, but you also increase the variance of the result.
The "accuracy" is increased because any bias for white or black is eliminated. By playing more and more games you will ultimately converge to the "correct" result.
The "variance" is larger because the 2 games from the same starting position will not be completely independent. This means that you'll need to play more games to reduce the uncertainty margins.

Robert

hi Robert,
What do you mean more precisely by that...completely independent ?

grts Bram

[Houdini]

hi Robert,
What do you mean more precisely by that...completely independent ?

grts Bram

If by accident an engine is well or poorly tuned for a given position/opening, it will show in both games played from the opening position. That means that the outcome of the two games is slightly correlated.

Cheers,
Robert

[Richard Allbert]

Do you have any idea about the 95% confidence interval on a 100-game match?

Robert

This is perhaps the worst, most arrogant response I've ever seen from a seller to a customer, no matter what the industry.

[Houdini]

This is not about "sellers" and "customers", strange that you try to add a commercial dimension to a technical discussion.

Robert

[Richard Allbert]

[Dr.Wael Deeb]

Do you have any idea about the 95% confidence interval on a 100-game match?

Robert

This is perhaps the worst, most arrogant response I've ever seen from a seller to a customer, no matter what the industry.

Only now you've noticed the arrogance of this guy

Although I must confess,his engine is a masterpiece....
Dr.D