Hi Don,
your move matching procedure is already great. But what do you think about the following proposal:
- Calculate the move match percentage also on a per-game level for all games with a minimum number of samples (i.e. moves not being ignored)
- List players according to their number of games where they exceeded a certain threshold of move matching
Also I would like to know where B. Ivanov occurs in your recent statistics, are there enough sample moves available for him?
Sven
move matching statistics
Moderators: hgm, Dann Corbit, Harvey Williamson
-
Sven
- Posts: 4052
- Joined: Thu May 15, 2008 9:57 pm
- Location: Berlin, Germany
- Full name: Sven Schüle
-
Don
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: move matching statistics
In the Zadar tournament where he played badly his match statistics are 35.7% out of 42 moves in the entire tournament:Sven Schüle wrote:Hi Don,
your move matching procedure is already great. But what do you think about the following proposal:
- Calculate the move match percentage also on a per-game level for all games with a minimum number of samples (i.e. moves not being ignored)
- List players according to their number of games where they exceeded a certain threshold of move matching
Also I would like to know where B. Ivanov occurs in your recent statistics, are there enough sample moves available for him?
Sven
29 35.71 15 42 4 Ivanov, Borislav
In the Veliko Tarnova he won, his match percentage was 67.31 out of a 52 game sample.
Please note that of 462 Grandmasters where the sample of moves is 52 or greater, not a single one has matched more then 64.10% of the time. So he is already higher than that.
Is that enough to be convincing? Not in my opinion. If this were the only evidence whatsoever it would not be nearly enough - it is only suspicious at best. Note that you are more likely to have a high match rate if you play less moves - and most of these players have played a lot more than 52 moves that can be sampled, so it's not particularly damning.
However, the other evidence against him is pretty powerful. If you strongly suspect someone of cheating based on other evidence and they you get an usually high match percentage as Borislav achieved, it really becomes highly suspicious.
I would like to find other tournaments that he competed in so that I could check those statistics too. But one must keep in mind that you cannot pick and choose what you test arbitrarily. For a test to be statistically valid it must be applied with some sort of consistent methodology. For example if I have 20 tournaments and I'm allowed to pick any two, I can probably get a high match rate by picking the 2 with the highest match rate.
You can calculate the odds of matching Houdini given a certain number of matchable moves in a single game using the 46% average hit rate. You could find the average for the player with the greatest hit rates and use that (perhaps with a margin added for safety) to calculate the odds of matching a single move and then calculate the odds of matching N moves using my testing methodology. Then we could run the test on any give game to count the matches and spit out a number which expresses the chances of this happening. This could be used to exonerate players who are suspected of cheating using Houdini - but based on what I am seeing here it would be difficult to use it to prove cheating.
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
Don
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: move matching statistics
It appears that Larry Kaufman is above average in his Houdini move matching statistics:
53.33 8 out of 15 1 games Kaufman,La
53.33 8 out of 15 1 games Kaufman,La
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
Don
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: move matching statistics
After processing 11239 games we find that on average a GM matches
Houdini 3 moves 45.71 percent of the time.
To have a meaningful test we need a way to match the moves of any
given game or contained set of games (such as in a single tournament)
and be able to estimate the likelihood of this happening.
One way to do this is by using binomial probabilities. This is valid
if the expected probablity of a match is exactly 50%. But as long as
we know the likelihood of a match of a single move we can calculate
for any given number of matches over some known number of moves what
the chances of a typical grandmaster making at least that number of
matches using other methods.
However, before just accepting the value of 45.71 percent for a match
rate it might be good to know if some players are significantly more
likely to match Houdini moves than other players.
It's difficult to determine that for any given player with a high
degree of accuracy without a significant number of samples. But based
on the samples have it appears that some players do indeed tend to
match Houdini more than other players. For example if I examine only
those players where I have at least 500 moves to sample, I get match
rates from 37.14% to 52.5%. 95 players had at least 500 moves to
sample.
If I bring the required number of samples down to 250 I have 253
players meeting that criteria. The match rates range from 37.14%
(same player as in the 500 move sample) to 54.41%. So it starts to
look as if something like 55% is reasonable upper bound on the move
match percentage of even the most Houdini-like human player.
For a good test we want to use very conservative values, so I next
checked players with at least 100 move samples and got 532 of those.
The top matching player matched 59.84 percent of Houdini's moves. So
from now on I am using 60% match rate as a kind of upper bound on what
we expect any player to match naturally (without referencing Houdini.)
For reference I also did players with 50 move samples and out of 713
players we had one who matched 60.78% of the moves of Houdini. This
player had 51 moves to match or 31 out of 51 = 60.78%. This gives me
a great deal of confidence that 60% is a very reasonable upper bound.
So to summarize, the average match rate is 45.71% for Grandmasters and
of the players with significant samples (say over 250 moves) the very
best match rate is 54.41%. If I allow tiny samples of 50 or more,
only 1 player exceeds 60% with a score of 60.78%. So I am taking 60%
as the maximum expected match percentage for even the most
Houdini-like player even though I believe the true average is no more
then 55%.
Note that this is not particularly scientific, it is more
observational and is based on only 11,239 games - there is no way to
know if some future player will come along who consistently matches
Houdini 65% of the time or more.
So using 60% as the upper bound I can calculate the odds of a player
matching Houdini N times out of M moves. Instead of wading through
the tedious math of probability calculation it was easier to run Monte
Carlo simulations to achieve an estimate with good accuracy.
So I wrote the program to do this using the 60% value and you provide
it with the number of moves and it will produce a table showing the
odds of matching these moves N times. For example if the sample is 10
moves I get this table:
This table implies that there is less than 1 percent chance that a
player will match Houdini's move 10 out 10 times in any given game.
There is almost a 5 percent chance he would match Houdini 9 out of 10
times in a 10 move sample, and so on according to the table. Again,
please note that I believe the odds are considerably lower - this is
the odds for a player who is highly inclined to match Houdini. But
for the average GM it would be far lower than 1%.
As a sanity check I looked at EVERY GAME where a player had exactly 10
moves to match (using the criteria of the tester) and there were 490
games. Not a single game showed a 10 out of 10 match but there were 5
games that showed a 9 out of 10 match. This is a good start because
the table indicates I should get almost 3 players with perfect 10 out
10 scores (out of 490 games.)
So I ran games with at least 10 move samples. Here are the top 20:
Running the table on 18 move samples you see that the odds of matching
18 out 18 as Papp,Ga did here is 0.010 percent. It's possible that
this game involved many forcing moves or perhaps simply was a game
composed of very easy to find best moves. But it does stand out as a
ridiculous exception and no other single games comes even close. On
the other hand, in a database with hundreds of thousands of games you
should actually expect to see a few of these. I will probably dig up
that game and post it as it may contain some clue in how to improve
the test.
I looked at Papp,Ga over all the games I have computed so far and I
only have 10 games of his to sample with an average match rate of
52.88%. For anyone interested here are the statistics for all 10
games he played:
If I check all 11,000 games using the tester, I find only 3 games where a
players probability of achieving the match rate was less than 1% which is quite
remarkable when you consider that over 11,000 games were tested. This is a
strong indication that my 60% value is very conservative.
Houdini 3 moves 45.71 percent of the time.
To have a meaningful test we need a way to match the moves of any
given game or contained set of games (such as in a single tournament)
and be able to estimate the likelihood of this happening.
One way to do this is by using binomial probabilities. This is valid
if the expected probablity of a match is exactly 50%. But as long as
we know the likelihood of a match of a single move we can calculate
for any given number of matches over some known number of moves what
the chances of a typical grandmaster making at least that number of
matches using other methods.
However, before just accepting the value of 45.71 percent for a match
rate it might be good to know if some players are significantly more
likely to match Houdini moves than other players.
It's difficult to determine that for any given player with a high
degree of accuracy without a significant number of samples. But based
on the samples have it appears that some players do indeed tend to
match Houdini more than other players. For example if I examine only
those players where I have at least 500 moves to sample, I get match
rates from 37.14% to 52.5%. 95 players had at least 500 moves to
sample.
If I bring the required number of samples down to 250 I have 253
players meeting that criteria. The match rates range from 37.14%
(same player as in the 500 move sample) to 54.41%. So it starts to
look as if something like 55% is reasonable upper bound on the move
match percentage of even the most Houdini-like human player.
For a good test we want to use very conservative values, so I next
checked players with at least 100 move samples and got 532 of those.
The top matching player matched 59.84 percent of Houdini's moves. So
from now on I am using 60% match rate as a kind of upper bound on what
we expect any player to match naturally (without referencing Houdini.)
For reference I also did players with 50 move samples and out of 713
players we had one who matched 60.78% of the moves of Houdini. This
player had 51 moves to match or 31 out of 51 = 60.78%. This gives me
a great deal of confidence that 60% is a very reasonable upper bound.
So to summarize, the average match rate is 45.71% for Grandmasters and
of the players with significant samples (say over 250 moves) the very
best match rate is 54.41%. If I allow tiny samples of 50 or more,
only 1 player exceeds 60% with a score of 60.78%. So I am taking 60%
as the maximum expected match percentage for even the most
Houdini-like player even though I believe the true average is no more
then 55%.
Note that this is not particularly scientific, it is more
observational and is based on only 11,239 games - there is no way to
know if some future player will come along who consistently matches
Houdini 65% of the time or more.
So using 60% as the upper bound I can calculate the odds of a player
matching Houdini N times out of M moves. Instead of wading through
the tedious math of probability calculation it was easier to run Monte
Carlo simulations to achieve an estimate with good accuracy.
So I wrote the program to do this using the 60% value and you provide
it with the number of moves and it will produce a table showing the
odds of matching these moves N times. For example if the sample is 10
moves I get this table:
Code: Select all
10 0.00602 0.602 (chance of matching AT LEAST 10 moves)
9 0.04032 4.633 (chance of matching AT LEAST 9 moves)
8 0.12096 16.729 (chance of matching AT LEAST 8 moves)
7 0.21512 38.241 (chance of matching AT LEAST 7 moves)
6 0.25080 63.321 (chance of matching AT LEAST 6 moves)
5 0.20048 83.369 (chance of matching AT LEAST 5 moves)
4 0.11153 94.522 (chance of matching AT LEAST 4 moves)
3 0.04246 98.768 (chance of matching AT LEAST 3 moves)
2 0.01063 99.830 (chance of matching AT LEAST 2 moves)
1 0.00159 99.989 (chance of matching AT LEAST 1 moves)
0 0.00011 100.000 (chance of matching AT LEAST 0 moves)
player will match Houdini's move 10 out 10 times in any given game.
There is almost a 5 percent chance he would match Houdini 9 out of 10
times in a 10 move sample, and so on according to the table. Again,
please note that I believe the odds are considerably lower - this is
the odds for a player who is highly inclined to match Houdini. But
for the average GM it would be far lower than 1%.
As a sanity check I looked at EVERY GAME where a player had exactly 10
moves to match (using the criteria of the tester) and there were 490
games. Not a single game showed a 10 out of 10 match but there were 5
games that showed a 9 out of 10 match. This is a good start because
the table indicates I should get almost 3 players with perfect 10 out
10 scores (out of 490 games.)
So I ran games with at least 10 move samples. Here are the top 20:
Code: Select all
100.00 18 18 Papp,Ga 62nd ch-HUN 2012
92.31 12 13 Nabaty,T ZMDI Open A 2012
92.31 12 13 Gavrilov,Al1 23rd Czech Open A
90.91 10 11 Rogozenco,D Bundesliga 2012-13
90.00 9 10 Romanov,E Botvinnik Memorial 2012
90.00 9 10 Michalik,P Groningen Open A
90.00 9 10 Li Chao2 2nd Indonesia Open
90.00 9 10 Dervishi,E 40th Olympiad Open
90.00 9 10 Berkes,F 62nd ch-HUN 2012
88.89 16 18 Wang Hao 1st FIDE GP London 2012
88.24 15 17 Edouard,R 45th Biel Masters Open
87.50 21 24 Kamsky,G SportAccord Blindfold Men 2012
87.50 14 16 Grigoryan,Av Chigorin Memorial
85.71 12 14 Lenic,L TCh-HUN 2012-13
85.71 12 14 Banusz,T 62nd ch-HUN 2012
84.62 11 13 Williams,Si1 4th London Classic Open
84.62 11 13 Nyback,T 40th Olympiad Open
84.62 11 13 Nikolov,Mo 40th Olympiad Open
84.62 11 13 Negi,P AICF-AAI Cup 2012
84.62 11 13 Ding Liren 6th SPICE Cup GM
18 out 18 as Papp,Ga did here is 0.010 percent. It's possible that
this game involved many forcing moves or perhaps simply was a game
composed of very easy to find best moves. But it does stand out as a
ridiculous exception and no other single games comes even close. On
the other hand, in a database with hundreds of thousands of games you
should actually expect to see a few of these. I will probably dig up
that game and post it as it may contain some clue in how to improve
the test.
I looked at Papp,Ga over all the games I have computed so far and I
only have 10 games of his to sample with an average match rate of
52.88%. For anyone interested here are the statistics for all 10
games he played:
Code: Select all
100.00 18 18 Papp,Ga 62nd ch-HUN 2012
75.00 3 4 Papp,Ga 62nd ch-HUN 2012
54.55 6 11 Papp,Ga 62nd ch-HUN 2012
50.00 7 14 Papp,Ga 62nd ch-HUN 2012
50.00 1 2 Papp,Ga TCh-SVK Extraliga 2012-13
46.67 7 15 Papp,Ga 62nd ch-HUN 2012
37.50 3 8 Papp,Ga 62nd ch-HUN 2012
35.71 5 14 Papp,Ga 62nd ch-HUN 2012
33.33 2 6 Papp,Ga 62nd ch-HUN 2012
25.00 3 12 Papp,Ga TCh-HUN 2012-13
players probability of achieving the match rate was less than 1% which is quite
remarkable when you consider that over 11,000 games were tested. This is a
strong indication that my 60% value is very conservative.
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
Sven
- Posts: 4052
- Joined: Thu May 15, 2008 9:57 pm
- Location: Berlin, Germany
- Full name: Sven Schüle
Re: move matching statistics
Hi Don,Don wrote:After processing 11239 games we find that on average a GM matches Houdini 3 moves 45.71 percent of the time.
[...]
As a sanity check I looked at EVERY GAME where a player had exactly 10
moves to match (using the criteria of the tester) and there were 490
games. Not a single game showed a 10 out of 10 match but there were 5
games that showed a 9 out of 10 match. This is a good start because
the table indicates I should get almost 3 players with perfect 10 out
10 scores (out of 490 games.)
So I ran games with at least 10 move samples. Here are the top 20:[...]Code: Select all
100.00 18 18 Papp,Ga 62nd ch-HUN 2012 92.31 12 13 Nabaty,T ZMDI Open A 2012 92.31 12 13 Gavrilov,Al1 23rd Czech Open A 90.91 10 11 Rogozenco,D Bundesliga 2012-13 90.00 9 10 Romanov,E Botvinnik Memorial 2012 90.00 9 10 Michalik,P Groningen Open A 90.00 9 10 Li Chao2 2nd Indonesia Open 90.00 9 10 Dervishi,E 40th Olympiad Open 90.00 9 10 Berkes,F 62nd ch-HUN 2012 88.89 16 18 Wang Hao 1st FIDE GP London 2012 88.24 15 17 Edouard,R 45th Biel Masters Open 87.50 21 24 Kamsky,G SportAccord Blindfold Men 2012 87.50 14 16 Grigoryan,Av Chigorin Memorial 85.71 12 14 Lenic,L TCh-HUN 2012-13 85.71 12 14 Banusz,T 62nd ch-HUN 2012 84.62 11 13 Williams,Si1 4th London Classic Open 84.62 11 13 Nyback,T 40th Olympiad Open 84.62 11 13 Nikolov,Mo 40th Olympiad Open 84.62 11 13 Negi,P AICF-AAI Cup 2012 84.62 11 13 Ding Liren 6th SPICE Cup GM
these are very interesting numbers. Can you also give those numbers for the games of B. Ivanov, with the condition "at least 10 move samples per game", not "exactly"? I do not want to enter the "cheating" discussion here, I would just like to know the available data.
Sven
-
Don
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: move matching statistics
I did run the one tournament using a version of the tool that only gives the combined statistics of all the games for a given player. I commented on that in a post in the general forum.Sven Schüle wrote:Hi Don,Don wrote:After processing 11239 games we find that on average a GM matches Houdini 3 moves 45.71 percent of the time.
[...]
As a sanity check I looked at EVERY GAME where a player had exactly 10
moves to match (using the criteria of the tester) and there were 490
games. Not a single game showed a 10 out of 10 match but there were 5
games that showed a 9 out of 10 match. This is a good start because
the table indicates I should get almost 3 players with perfect 10 out
10 scores (out of 490 games.)
So I ran games with at least 10 move samples. Here are the top 20:[...]Code: Select all
100.00 18 18 Papp,Ga 62nd ch-HUN 2012 92.31 12 13 Nabaty,T ZMDI Open A 2012 92.31 12 13 Gavrilov,Al1 23rd Czech Open A 90.91 10 11 Rogozenco,D Bundesliga 2012-13 90.00 9 10 Romanov,E Botvinnik Memorial 2012 90.00 9 10 Michalik,P Groningen Open A 90.00 9 10 Li Chao2 2nd Indonesia Open 90.00 9 10 Dervishi,E 40th Olympiad Open 90.00 9 10 Berkes,F 62nd ch-HUN 2012 88.89 16 18 Wang Hao 1st FIDE GP London 2012 88.24 15 17 Edouard,R 45th Biel Masters Open 87.50 21 24 Kamsky,G SportAccord Blindfold Men 2012 87.50 14 16 Grigoryan,Av Chigorin Memorial 85.71 12 14 Lenic,L TCh-HUN 2012-13 85.71 12 14 Banusz,T 62nd ch-HUN 2012 84.62 11 13 Williams,Si1 4th London Classic Open 84.62 11 13 Nyback,T 40th Olympiad Open 84.62 11 13 Nikolov,Mo 40th Olympiad Open 84.62 11 13 Negi,P AICF-AAI Cup 2012 84.62 11 13 Ding Liren 6th SPICE Cup GM
these are very interesting numbers. Can you also give those numbers for the games of B. Ivanov, with the condition "at least 10 move samples per game", not "exactly"? I do not want to enter the "cheating" discussion here, I would just like to know the available data.
Sven
However, I will run this again on all the individual game I current have available and get back to you.
Don
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
Sven
- Posts: 4052
- Joined: Thu May 15, 2008 9:57 pm
- Location: Berlin, Germany
- Full name: Sven Schüle
Re: move matching statistics
Thanks, that was exactly what I meant. I was aware of the "combined" statistics but now I am interested on the "per game" data as well.Don wrote:However, I will run this again on all the individual game I current have available and get back to you.
Sven
-
Don
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: move matching statistics
The format is a little sloppy, but here is the data I have. Personally I don't find any of it suspicious at all. I hope you can read it - if you have any questions let me know. This test is not very useful for single games in general unless there is a really a lot of samples and it's designed to be ultra paranoid about false positives.Sven Schüle wrote:Thanks, that was exactly what I meant. I was aware of the "combined" statistics but now I am interested on the "per game" data as well.Don wrote:However, I will run this again on all the individual game I current have available and get back to you.
Sven
Code: Select all
1 0.60008 60.008 (chance of matching AT LEAST 1 moves)
100.00 1 1 Ivanov, Borislav 18th Open B
9 0.14172 22.533 (chance of matching AT LEAST 9 moves)
75.00 9 12 Ivanov, Borislav Veliko Tarnovo BUL, 1st Open r_5
10 0.19270 64.042 (chance of matching AT LEAST 10 moves)
58.82 10 17 Deur, Zrinka 18th Open B
7 0.22723 66.528 (chance of matching AT LEAST 7 moves)
58.33 7 12 Kristovic, Marijan 18th Open B
4 0.29054 71.047 (chance of matching AT LEAST 4 moves)
57.14 4 7 Ivanov, Borislav Veliko Tarnovo BUL, 1st Open r_3
8 0.14157 85.771 (chance of matching AT LEAST 8 moves)
50.00 8 16 Ivanov, Borislav Veliko Tarnovo BUL, 1st Open r_7
7 0.08399 94.169 (chance of matching AT LEAST 7 moves)
43.75 7 16 Ivanov, Borislav 18th Open B
10 0.03186 97.833 (chance of matching AT LEAST 10 moves)
41.67 10 24 Rombaldoni, Axel Veliko Tarnovo BUL, 1st Open r_7
4 0.11153 94.528 (chance of matching AT LEAST 4 moves)
40.00 4 10 Nancheva, Doroteya Veliko Tarnovo BUL, 1st Open r_1
5 0.06554 96.795 (chance of matching AT LEAST 5 moves)
38.46 5 13 Stanojoski, Zvonko Veliko Tarnovo BUL, 1st Open r_5
4 0.06992 97.074 (chance of matching AT LEAST 4 moves)
36.36 4 11 Ivanov, Borislav Veliko Tarnovo BUL, 1st Open r_1
4 0.00399 99.906 (chance of matching AT LEAST 4 moves)
25.00 4 16 Ivanov, Borislav 18th Open B
2 0.04111 99.143 (chance of matching AT LEAST 2 moves)
25.00 2 8 Maciol, Ryszard Veliko Tarnovo BUL, 1st Open r_3
1 0.15354 97.441 (chance of matching AT LEAST 1 moves)
25.00 1 4 Ivanov, Borislav Veliko Tarnovo BUL, 1st Open r_2
1 0.07675 98.978 (chance of matching AT LEAST 1 moves)
20.00 1 5 Vidovic, Ante 18th Open B
0 0.02566 100.000 (chance of matching AT LEAST 0 moves)
0.00 0 4 Malinov, Boyan Veliko Tarnovo BUL, 1st Open r_2
0 0.02566 100.000 (chance of matching AT LEAST 0 moves)
0.00 0 4 Ivanov, Borislav 18th Open B
0 0.39985 100.000 (chance of matching AT LEAST 0 moves)
0.00 0 1 Glavicic, Tino 18th Open B
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
Sven
- Posts: 4052
- Joined: Thu May 15, 2008 9:57 pm
- Location: Berlin, Germany
- Full name: Sven Schüle
Re: move matching statistics
Well done, I can read it, thanks againDon wrote:The format is a little sloppy, but here is the data I have. Personally I don't find any of it suspicious at all. I hope you can read it - if you have any questions let me know. This test is not very useful for single games in general unless there is a really a lot of samples and it's designed to be ultra paranoid about false positives.
Here all rated games that have been played by B. Ivanov in the past ~5 years (currently ending in feb-2013) are listed including the PGNs. It might be interesting to analyze these with your tool to check whether there is some "smooth increase" in the Houdini move match that might be the result of computer-based training, whether the Houdini move match was already high in some earlier games of B.I., or whether there is some spontaneous jump in certain recent games or tournaments. Do you think that is possible?
It would also show the actual data for the Zadar 2012 tournament which might be of high interest to some.
Sven
-
mvk
- Posts: 589
- Joined: Tue Jun 04, 2013 10:15 pm
Thirteen sigma
http://rjlipton.wordpress.com/2013/07/2 ... een-sigma/ suggests a complete tournament was faked. Can you run those games through your tester as well?