Chess Statistics

[Milos]

Seems you do not understand the result. I know that it is a trinomial distribution, used to calculate the error intervals, there the number of draws is very important. But for LOS it is not. Can you understand that or your brain is just flat?

What Edmund is saying is correct, what you are saying simply has no sense at all. You provided no argument at all why ignore draws. Why would you for calculating error bars use one distribution, and for calculating LOS another???
Unless your definition of LOS is likelihood of superiority when draws are ignored.

[Laskos]

Seems you do not understand the result. I know that it is a trinomial distribution, used to calculate the error intervals, there the number of draws is very important. But for LOS it is not. Can you understand that or your brain is just flat?

What Edmund is saying is correct, what you are saying simply has no sense at all. You provided no argument at all why ignore draws. Why would you for calculating error bars use one distribution, and for calculating LOS another???
Unless your definition of LOS is likelihood of superiority when draws are ignored.

If you do not know what you are saying, perform Monte Carlo, and you will see that numbers of draws does not count. Even the statistical formula provided here shows pretty much that, as it is very insensitive to the number of draws, the sensitivity is an artifact of presumed normal distribution. If you sum and integrate for LOS with an initial trinomial, number of draws cancels in the distribution. I hope you do not intend LOS as a result of a game, but a result of a match.

Put some numbers in the precise formula, then check with the approximate statistical result.

Kai

[Edmund]

Seems you do not understand the result. I know that it is a trinomial distribution, used to calculate the error intervals, there the number of draws is very important. But for LOS it is not. Can you understand that or your brain is just flat?

What Edmund is saying is correct, what you are saying simply has no sense at all. You provided no argument at all why ignore draws. Why would you for calculating error bars use one distribution, and for calculating LOS another???
Unless your definition of LOS is likelihood of superiority when draws are ignored.

If you do not know what you are saying, perform Monte Carlo, and you will see that numbers of draws does not count. Even the statistical formula provided here shows pretty much that, as it is very insensitive to the number of draws, the sensitivity is an artifact of presumed normal distribution. If you sum and integrate for LOS with an initial trinomial, number of draws cancels in the distribution. I hope you do not intend LOS as a result of a game, but a result of a match.

Put some numbers in the precise formula, then check with the approximate statistical result.

Kai

When we last spoke about this topic. We developed an algorithm using the trinomial distribution. With this program I calculated a LOS Table, I later uploaded to http://chessprogramming.wikispaces.com/LOS+Table (note that for this table I assumed a draw ratio of 0.32) This table correlates very much with some figures I calculated around the same time with a monte carlo algorithm, though I didn't record the findings. But you can compare the table with the one published by Ciarrochi http://www.husvankempen.de/nunn/rating/tablejoseph.htm, who used monte carlo himself and you will find that the values are equal.

I then rewrote the algorithm to use the normal distribution and I was able to reproduce the table. That is the Normal distribution with a
mean = N/2 and
sd = N(1-draw_ratio).

Using any other sd you are not able to reproduce the table. So the draw_ratio is very much important here.

And also if you think about it intuitively, being ahead a constant amount of points is harder against a player who is drawing a lot vs one against whom all games are decided.

[Laskos]

Seems you do not understand the result. I know that it is a trinomial distribution, used to calculate the error intervals, there the number of draws is very important. But for LOS it is not. Can you understand that or your brain is just flat?

What Edmund is saying is correct, what you are saying simply has no sense at all. You provided no argument at all why ignore draws. Why would you for calculating error bars use one distribution, and for calculating LOS another???
Unless your definition of LOS is likelihood of superiority when draws are ignored.

If you do not know what you are saying, perform Monte Carlo, and you will see that numbers of draws does not count. Even the statistical formula provided here shows pretty much that, as it is very insensitive to the number of draws, the sensitivity is an artifact of presumed normal distribution. If you sum and integrate for LOS with an initial trinomial, number of draws cancels in the distribution. I hope you do not intend LOS as a result of a game, but a result of a match.

Put some numbers in the precise formula, then check with the approximate statistical result.

Kai

When we last spoke about this topic. We developed an algorithm using the trinomial distribution. With this program I calculated a LOS Table, I later uploaded to http://chessprogramming.wikispaces.com/LOS+Table (note that for this table I assumed a draw ratio of 0.32) This table correlates very much with some figures I calculated around the same time with a monte carlo algorithm, though I didn't record the findings. But you can compare the table with the one published by Ciarrochi http://www.husvankempen.de/nunn/rating/tablejoseph.htm, who used monte carlo himself and you will find that the values are equal.

I then rewrote the algorithm to use the normal distribution and I was able to reproduce the table. That is the Normal distribution with a
mean = N/2 and
sd = N(1-draw_ratio).

Using any other sd you are not able to reproduce the table. So the draw_ratio is very much important here.

And also if you think about it intuitively, being ahead a constant amount of points is harder against a player who is drawing a lot vs one against whom all games are decided.

I do not want to enter into intuitive guesses. The draws for LOS do not enter in the precise calculation of LOS given by my earlier PRECISE formula. It cancels. If you go to check it, you will see that a statistical calculation (I have one, but I do not know how to attach it to a message or PM, it is an .exe file), but you will see that the result for LOS is very insensitive to draws, even in these, assuming normal distribution formulas. Try 1100, 100, 900, to 1100, 1000, 900, it varies less by 1% in my STATISTICAL calculation.

Take the PRECISE formula as a rule for LOS, independent of draws. If you want my STATISTICAL (as yours) program, which calculates Standard Deviations, LOS, number of games needed for LOS 0.80(0.20), 0.95(0.05) and 0.99(0.01) send me your e-mail by PM or something.

But I repeat, LOS does not depend on the number of draws. Standard Deviation depends heavily on the number of draws, as it is given, on average, for pretty normal distributions as sqrt(score*(1-score) - 0.25*DrawFraction)/sqrt(NumberGames). But even this is an approximation, which works well with more number of games and not very skewed results, not, say, with 95:3:2, where normal distribution fails.

Kai

[Dann Corbit]

you will notice that the draw rate is very much considered.

It shouldn't be considered for LOS.
(your example is wrong, look at the number of losses).

Seams you have a different definition of LOS. How are you defining it then?

P.S. Table formulas like ones based on just difference in wins/losses are simply wrong.

Likelihood of Success that one engine is better than another. It does not depend on the number of draws. Error intervals yes, depend. If you want a precise formula for LOS, I can give it.

Kai

Something seems wrong in this idea.
A and B play a one million game match, with 3 wins for A and the rest draws.

C and D play a three game match with 3 wins for C.

I think that the LOS is very different for these two.

[Laskos]

you will notice that the draw rate is very much considered.

It shouldn't be considered for LOS.
(your example is wrong, look at the number of losses).

Seams you have a different definition of LOS. How are you defining it then?

P.S. Table formulas like ones based on just difference in wins/losses are simply wrong.

Likelihood of Success that one engine is better than another. It does not depend on the number of draws. Error intervals yes, depend. If you want a precise formula for LOS, I can give it.

Kai

Something seems wrong in this idea.
A and B play a one million game match, with 3 wins for A and the rest draws.

C and D play a three game match with 3 wins for C.

I think that the LOS is very different for these two.

It is NOT an idea, it's a hard fact. The PRECISE formula for LOS does not include draws. Check by whatever means you have (correct ones).

Kai

[Milos]

I do not want to enter into intuitive guesses. The draws for LOS do not enter in the precise calculation of LOS given by my earlier PRECISE formula. It cancels. If you go to check it, you will see that a statistical calculation (I have one, but I do not know how to attach it to a message or PM, it is an .exe file), but you will see that the result for LOS is very insensitive to draws, even in these, assuming normal distribution formulas. Try 1100, 100, 900, to 1100, 1000, 900, it varies less by 1% in my STATISTICAL calculation.

But I repeat, LOS does not depend on the number of draws. Standard Deviation depends heavily on the number of draws, as it is given, on average, for pretty normal distributions as sqrt(score*(1-score) - 0.25*DrawFraction)/sqrt(NumberGames). But even this is an approximation, which works well with more number of games and not very skewed results, not, say, with 95:3:2, where normal distribution fails.

LOS does depend on number of draws, but not much. And in normal conditions it is not possible to detect the difference.
In formula for sigma that is Edmund using it really cancels out. But the reason for this is that the formula is only an approximation.
In his formula (w, d, l and N are number of wins, draws, losses and games, respectively):
LOS=Phi(x), where x=(w-l)/sqrt(N-d)=(w-l)/sqrt(w+l) where x really does not depend on d.
However, accurately
x=(mu-0.5)/sigma=(w-l)/(2N*sigma)=...=(w-l)/sqrt((2w+d)(2l+d)/N-d)=(w-l)/sqrt(N-d-((w-l)/N)^2)

So, Edumund takes for his factor w+l while accurate value is w+l-((w-l)/n)^2. The difference is not significant, but it's not zero.
The more the games are played, and the less is the difference between engines, the less is the impact of number of draws on LOS.

[Laskos]

I do not want to enter into intuitive guesses. The draws for LOS do not enter in the precise calculation of LOS given by my earlier PRECISE formula. It cancels. If you go to check it, you will see that a statistical calculation (I have one, but I do not know how to attach it to a message or PM, it is an .exe file), but you will see that the result for LOS is very insensitive to draws, even in these, assuming normal distribution formulas. Try 1100, 100, 900, to 1100, 1000, 900, it varies less by 1% in my STATISTICAL calculation.

But I repeat, LOS does not depend on the number of draws. Standard Deviation depends heavily on the number of draws, as it is given, on average, for pretty normal distributions as sqrt(score*(1-score) - 0.25*DrawFraction)/sqrt(NumberGames). But even this is an approximation, which works well with more number of games and not very skewed results, not, say, with 95:3:2, where normal distribution fails.

LOS does depend on number of draws, but not much. And in normal conditions it is not possible to detect the difference.
In formula for sigma that is Edmund using it really cancels out. But the reason for this is that the formula is only an approximation.
In his formula (w, d, l and N are number of wins, draws, losses and games, respectively):
LOS=Phi(x), where x=(w-l)/sqrt(N-d)=(w-l)/sqrt(w+l) where x really does not depend on d.
However, accurately
x=(mu-0.5)/sigma=(w-l)/(2N*sigma)=...=(w-l)/sqrt((2w+d)(2l+d)/N-d)=(w-l)/sqrt(N-d-((w-l)/N)^2)

So, Edumund takes for his factor w+l while accurate value is w+l-((w-l)/n)^2. The difference is not significant, but it's not zero.
The more the games are played, and the less is the difference between engines, the less is the impact of number of draws on LOS.

Impact of draws = 0 in PRECISE LOS which I gave, not statistical one, with standard deviations and normal distributions. Please check by other methods. I also used statistical programs built in several hours by me, there the dependence appears slightly, but when I got the PRECISE result, I saw that the differences depending on the draws were an artifact of normal distribution assumption.

Kai

[Edmund]

I do not want to enter into intuitive guesses. The draws for LOS do not enter in the precise calculation of LOS given by my earlier PRECISE formula. It cancels. If you go to check it, you will see that a statistical calculation (I have one, but I do not know how to attach it to a message or PM, it is an .exe file), but you will see that the result for LOS is very insensitive to draws, even in these, assuming normal distribution formulas. Try 1100, 100, 900, to 1100, 1000, 900, it varies less by 1% in my STATISTICAL calculation.

But I repeat, LOS does not depend on the number of draws. Standard Deviation depends heavily on the number of draws, as it is given, on average, for pretty normal distributions as sqrt(score*(1-score) - 0.25*DrawFraction)/sqrt(NumberGames). But even this is an approximation, which works well with more number of games and not very skewed results, not, say, with 95:3:2, where normal distribution fails.

LOS does depend on number of draws, but not much. And in normal conditions it is not possible to detect the difference.
In formula for sigma that is Edmund using it really cancels out. But the reason for this is that the formula is only an approximation.
In his formula (w, d, l and N are number of wins, draws, losses and games, respectively):
LOS=Phi(x), where x=(w-l)/sqrt(N-d)=(w-l)/sqrt(w+l) where x really does not depend on d.
However, accurately
x=(mu-0.5)/sigma=(w-l)/(2N*sigma)=...=(w-l)/sqrt((2w+d)(2l+d)/N-d)=(w-l)/sqrt(N-d-((w-l)/N)^2)

So, Edumund takes for his factor w+l while accurate value is w+l-((w-l)/n)^2. The difference is not significant, but it's not zero.
The more the games are played, and the less is the difference between engines, the less is the impact of number of draws on LOS.

Impact of draws = 0 in PRECISE LOS which I gave, not statistical one, with standard deviations and normal distributions. Please check by other methods. I also used statistical programs built in several hours by me, there the dependence appears slightly, but when I got the PRECISE result, I saw that the differences depending on the draws were an artifact of normal distribution assumption.

Kai

Lets calculate an example and you tell me please where I am going wrong:

Test A: 1/1/0
Test B: 1/2/0

A)
win_prob = 1 / 4
draw_prob = 1 / 2

P(score_dif = 0) = N! / (W! D! L!) win_prob^W draw_prob^D win_prob^L
= 2! / (0! 2! 0!) 0.25^0 * 0.5^2 * 0.25^0 = 0.5^2 = 0.25
P(score_dif = 1) = 2! / (1! 1! 0!) 0.25^1 * 0.5^1 * 0.25^0 = 2 * 0.25 * 0.5 = 0.25

LOS ( A ) = 1 - (P(score_dif = 0) + P(score_dif = 1) /2) = 62.5%

B)
win_prob = 1 / 6
draw_prob = 2 / 3

P(score_dif = 0) = 3! / (0! 3! 0!) (1/6)^0 * (2/3)^3 * (1/6)^0 = (2/3)^3 = 0.296296296
P(score_dif = 1) = 3! / (1! 2! 0!) (1/6)^1 * (2/3)^2 * (1/6)^0 = (1/6) * (2/3)^2 = 0.222222222

LOS ( B ) = 1 - (P(score_dif = 0) + P(score_dif = 1) /2) = 59.26%

---

So the LOS for the Test B is lower although the score difference is equal.

[Laskos]

I do not want to enter into intuitive guesses. The draws for LOS do not enter in the precise calculation of LOS given by my earlier PRECISE formula. It cancels. If you go to check it, you will see that a statistical calculation (I have one, but I do not know how to attach it to a message or PM, it is an .exe file), but you will see that the result for LOS is very insensitive to draws, even in these, assuming normal distribution formulas. Try 1100, 100, 900, to 1100, 1000, 900, it varies less by 1% in my STATISTICAL calculation.

But I repeat, LOS does not depend on the number of draws. Standard Deviation depends heavily on the number of draws, as it is given, on average, for pretty normal distributions as sqrt(score*(1-score) - 0.25*DrawFraction)/sqrt(NumberGames). But even this is an approximation, which works well with more number of games and not very skewed results, not, say, with 95:3:2, where normal distribution fails.

LOS does depend on number of draws, but not much. And in normal conditions it is not possible to detect the difference.
In formula for sigma that is Edmund using it really cancels out. But the reason for this is that the formula is only an approximation.
In his formula (w, d, l and N are number of wins, draws, losses and games, respectively):
LOS=Phi(x), where x=(w-l)/sqrt(N-d)=(w-l)/sqrt(w+l) where x really does not depend on d.
However, accurately
x=(mu-0.5)/sigma=(w-l)/(2N*sigma)=...=(w-l)/sqrt((2w+d)(2l+d)/N-d)=(w-l)/sqrt(N-d-((w-l)/N)^2)

So, Edumund takes for his factor w+l while accurate value is w+l-((w-l)/n)^2. The difference is not significant, but it's not zero.
The more the games are played, and the less is the difference between engines, the less is the impact of number of draws on LOS.

Impact of draws = 0 in PRECISE LOS which I gave, not statistical one, with standard deviations and normal distributions. Please check by other methods. I also used statistical programs built in several hours by me, there the dependence appears slightly, but when I got the PRECISE result, I saw that the differences depending on the draws were an artifact of normal distribution assumption.

Kai

Lets calculate an example and you tell me please where I am going wrong:

Test A: 1/1/0
Test B: 1/2/0

A)
win_prob = 1 / 4
draw_prob = 1 / 2

P(score_dif = 0) = N! / (W! D! L!) win_prob^W draw_prob^D win_prob^L
= 2! / (0! 2! 0!) 0.25^0 * 0.5^2 * 0.25^0 = 0.5^2 = 0.25
P(score_dif = 1) = 2! / (1! 1! 0!) 0.25^1 * 0.5^1 * 0.25^0 = 2 * 0.25 * 0.5 = 0.25

LOS ( A ) = 1 - (P(score_dif = 0) + P(score_dif = 1) /2) = 62.5%

B)
win_prob = 1 / 6
draw_prob = 2 / 3

P(score_dif = 0) = 3! / (0! 3! 0!) (1/6)^0 * (2/3)^3 * (1/6)^0 = (2/3)^3 = 0.296296296
P(score_dif = 1) = 3! / (1! 2! 0!) (1/6)^1 * (2/3)^2 * (1/6)^0 = (1/6) * (2/3)^2 = 0.222222222

LOS ( B ) = 1 - (P(score_dif = 0) + P(score_dif = 1) /2) = 59.26%

---

So the LOS for the Test B is lower although the score difference is equal.

Sorry, I didn't quite follow your win_prob and draw_prob. I understood that 2 games were played in A) and 3 in B). Besides that, your formula for LOS already assumes normal distribution. Try to check with the exact formula for LOS.

Kai