Borislav Ivanov: a Lilov's add-on

[mwyoung]

Code: Select all

match perc   matches   samples  games  player 
 ----------  --------  --------  -----  ----------------- 
      77.55        38        49      1  Magnus Carlsen 
      71.43        35        49      1  Teimour Radjabov

For the some game using the Lilov-Houdart testing protocol GM Carlsen scored 94.5%.

Remember the theory is no human can play a game of chess like a computer. This is not my assertion Don, and it is not my testing protocol.

And it's not mine either and I have written before that I have not looked at their data to check it for myself and now that I know how they are doing it I completely disagree with their methodology.

No matter what methodology is used, including mine, you cannot just run the test on one game or games of one person - you have to test games from many tournaments and individuals in order to establish what is normal and what is not.

I think we can all agree that a reasonable player is going to match the moves of Houdini some percentage of time - but nobody really knows for sure what is outside the range of credulity. But we can do some studies to figure this out.

I couldn't care less about what Lilov is saying by the way, I like to see things for myself. I think it's very clear he is cheating even without the move matching but if it is to be properly used as evidence it needs to be done right.

We sometimes talk about innocent people being accused and convicted but the flip side of the coin is the number of guilty people who go free because the prosecutors were sloppy or made mistakes which cast doubts on all the evidence.

In the testing protocol used by FM Lilov no expections were made of any kind except for opening theory when move matching.

No game length was ever stated.

No amount of games was every stated.

If you watch all the video's you will understand why FM Lilov did not want to use a fair set of bench marks. FM Lilov thought BI was trying to hide the use of a computer at some points in the games, or in other games that BI played. FM Lilov did not want to show anything in his videos that FM Lilov did not think showed BI guilt.

FM Lilov setup this testing protocol to prove BI guilt in hindsight.

FM Lilov could not set up fair standards, because in some games BI had a low percentage of computer move matches. That FM Lilov would have to explain away those move and games.

FM Lilov said in his videos as one explanation for BI not move matching. When BI move matched the computer BI was cheating, when BI did not move match the computer BI either chose not to cheat, or BI cheating device malfunctioned.

I am not proclaiming if BI is guilty or not.

My problem is the testing protocol used, with no research of any kind to show it works by catching known cheaters, and not claiming guilt of innocent chess players.

I would hope no one would want FM Lilov's testing standard to be used. Because it was set up for the one reason. To do a hatchet job on BI. That makes for a poor testing protocol.

I don't have a problem with using some kind of testing protocol to detect cheating, and used as a tool. But it should be shown to work, and not flag chess players who are not cheating.

I agree with you 100% here. I always considered the move matching stuff a kind of distraction - a technique typically used by the defense, pick a fight with something minor and try to make it the core of the matter when it isn't.

I am glad you took the time to see FM Lilov's videos. And why I had my opinions, because like you I test things for myself also. Like I said before we have the same opinion on BI. But this is not about if BI is guilty or not. It is about how we say something is proof in a fair way.

Why is FM Lilov methodology and videos important in this discussion, and why FM Lilov's testing protocol flaws must be shown?

This is what was reported on Chessbase news regarding FM Lilov's videos and testing methodology, and why I was so outspoken against it. I don't want FM Lilov's testing protocol or any protocol that has not been tested and shown to work to by used against any chess player.

From Chessbase news.

FM Valeri Lilov’s extensive analysis of Ivanov’s games have played central role in the international investigation of Ivanov’s alleged cheating scheme. Many sources have refered back to Lilov’s YouTube video annotation of the alleged cheater’s latest games in Kustendil and Veliko Tarnovo.

[mwyoung]

I still have more games to run for this tournament, but here is what I have so far based on the games of the "Veliko Tarnovo BUL" - I'm not sure what tournament that is or how he did in that one. I sorted this by match percentage.

My table show how many moves matched there were out of how many samples and how many games were involved. I doubt my methodology is perfect but it's probably a whole lot better than tallying a match if it was one of several Houdini choices. I still cannot believe how silly that is.

It's not a large sample but at least we start to get a rough sense of what match percentage would be unusually high or low for any given player.

I'm pretty sure that even if you were getting advice from a computer and playing its first choice you would not get 100% match since you cannot be sure of the many variables involved, such as how much thinking time is allowed, the number of cores set and non-deterministic program behavior.

In the first game the move matching percentage was pretty low for Borislav, only 60% which is close to the median. The rest are quite high but I'm not showing individual games here. I will add that as a feature later to the script and I will test this again at a higher level of play for Houdini 3. I also would like to see how these same players do when matched against Stockfish or Komodo.

Code: Select all

match perc   matches   samples  games  player
----------  --------  --------  -----  -----------------
     69.61       126       181      6  Dimitrov, Radoslav
     69.09        38        55      4  Ivanov, Borislav
     68.15        92       135      4  Dimitrov, Dejan
     67.57        50        74      6  Stanojoski, Zvonko
     67.33        68       101      4  Nikolovski, Nikola
     66.67        74       111      6  Rombaldoni, Axel
     66.67        40        60      6  Galunov, Todor
     66.67         4         6      3  Petrov, Martin
     66.30        61        92      6  Katov, Luboslav
     65.47        91       139      5  Maciol, Ryszard
     65.00        39        60      2  Angelov, Angel Y
     64.38       103       160      6  Nevednichy, Boris M
     64.00        16        25      3  Iliev, Nikolay
     63.93        78       122      5  Nikolov, Sasho
     63.64        21        33      1  Vasilev, Martin
     63.41        52        82      4  Galunova, Tsveta
     62.81        76       121      6  Tashkov, Rumen
     62.73        69       110      4  Petkov, Emil
     62.71       111       177      5  Dimitrov, Pavel
     62.00        62       100      5  Drenchev, Petar
     61.48        83       135      6  Stamenkov, Vanco
     61.36        54        88      2  Ivanova, Simoneta
     61.29        38        62      3  Dereshki, Dario
     60.98        25        41      6  Angelov, Kosta
     60.00        18        30      1  Pencheva, Iva
     60.00         6        10      1  Stefanov, Evtim
     60.00         6        10      1  Mihtis, Theodoros
     59.76        98       164      6  Gazis, Efstathios
     59.49        47        79      5  Janev, Pavel
     59.21        45        76      3  Shentov, Petar-Delian
     58.97        23        39      1  Ilijkov, Ivan
     58.49        31        53      4  Ninov, Dayan
     58.39        87       149      6  Ivelinov, Hristo
     58.29       116       199      6  Erwich, Frank
     57.84        59       102      4  Veselinov, Dimcho
     57.58        38        66      2  Stoyanov, Ivaylo
     57.48        73       127      5  Senetia, Teodor
     57.14        24        42      1  Sisoev, Robert
     57.14         8        14      1  Nikolovska, Dragana
     56.86        29        51      3  Stefanov, Emil
     56.80        71       125      5  Kukov, Velislav
     56.10        23        41      2  Nikolov, Nikolay Petrov
     55.56        10        18      1  Kamenov, Valentin
     55.22        37        67      4  Petrov, Vladimir Sergeev
     55.00        33        60      3  Kazakov, Peter
     54.95        50        91      3  Milea, Florian
     54.43        86       158      5  Samu, Sorin-Mihai
     53.15        76       143      4  Kalchev, Stefan
     52.94         9        17      1  Nancheva, Doroteya
     52.63        20        38      2  Mahlev, Atanas
     52.63        10        19      1  Atanasov, Viktor Atanasov
     51.82        71       137      3  Ljangov, Petar
     51.47        35        68      2  Mitev, Miroslav
     51.22        21        41      2  Stoyanov, Valeri
     50.85        30        59      4  Bochev, Krasimir
     48.94        23        47      1  Lazarov, Janaki
     48.65        18        37      1  Lim, Kian Hwa
     48.48        16        33      3  Ilchev, Plamen
     48.11        51       106      3  Romcovici, Victor
     46.67        14        30      1  Sotirov, Ilia
     46.34        19        41      3  Marjanovics, Annamaria
     40.54        15        37      3  Marjanovics, Gyorgy
     38.89        14        36      1  Apostu, Toader
     33.33         3         9      1  Ivanov, Yordan Kr
     33.33         2         6      1  Malinov, Boyan
     33.33         1         3      1  Simeonov, Svetoslav
     30.00         3        10      1  Ivanov, Oleg
     28.95        11        38      3  Tzouvelekis, Ioannis
     28.57         2         7      1  Todorov, Petar G
     20.51         8        39      2  Kolev, Velio

Now we are talking. Keep the data coming, and the methods used. I love good data.

You should sell this application it could have many uses.

If you can test other tournaments like Tal 2013. I would like to know what the top players percentage for comparison.

I was curious who Dimitrov, Radoslav was.

FIDE Chess Profile

2908549 Dimitrov, Radoslav
Federation Bulgaria
FIDE title International Master
Rating:
std.
2423 rapid
2438 blitz
2448

B-Year 1993
Sex Male

Don you may find this useful regarding BI tournaments and what to test.

This is one of the tournaments "Veliko Tarnovo BUL" FM Lilov claimed BI was cheating. BI scored 8.0/9.0.

[Uri Blass]

Note that the main evidence of lilov that increased my confidence that ivanov is quilty is not the good moves of ivanov but the fact that there are also games that he played like a weak player the way that he lost against weak players.

I agree that I saw no statistical proof based on matchs with houdini in the analysis of lilov.

For don
I think that it may be better not to use move 16 and after it for research and it is better to use simply only positions that never happened earlier(that may be sometimes before move 15 and sometimes after move 15).

There can be some human preperation based on houdini before the game of positions that humans never played so this is not perfect and I had already one game when my opponent told me after the game that the evaluation of houdini is more than 6 pawns for himself in some non theory position(non theory at least based on the database that I have) and I believe him that he did not cheat and simply prepared at home in the specific line(and preparing something at home with houdini before the game is of course legal).

[sicilianquake87]

There can be some human preperation based on houdini before the game of positions that humans never played so this is not perfect and I had already one game when my opponent told me after the game that the evaluation of houdini is more than 6 pawns for himself in some non theory position(non theory at least based on the database that I have) and I believe him that he did not cheat and simply prepared at home in the specific line(and preparing something at home with houdini before the game is of course legal).

I think Lilov commented on this in a sarcastic way saying something like "he is a man of profound knowledge" and I must say I agree with him. This can happen in a particular line you analized at home, but cannot happen in the huge variety of the opening repertoire that BI displayed in the last year. It takes a hell lot of time to prepare well a single opening while this guy plays anything effortlessly without blunders or weak or dubious moves. It may well be that he is a genius and nobody noticed him but looking at the games, the opponents and the time in which such impressive player developed himself, it is higly questionable and improbable.

[Don]

Note that the main evidence of lilov that increased my confidence that ivanov is quilty is not the good moves of ivanov but the fact that there are also games that he played like a weak player the way that he lost against weak players.

I agree that I saw no statistical proof based on matchs with houdini in the analysis of lilov.

For don
I think that it may be better not to use move 16 and after it for research and it is better to use simply only positions that never happened earlier(that may be sometimes before move 15 and sometimes after move 15).

I would prefer that but it would require building a database to get this right and theory changes.

There can be some human preperation based on houdini before the game of positions that humans never played so this is not perfect and I had already one game when my opponent told me after the game that the evaluation of houdini is more than 6 pawns for himself in some non theory position(non theory at least based on the database that I have) and I believe him that he did not cheat and simply prepared at home in the specific line(and preparing something at home with houdini before the game is of course legal).

[mwyoung]

Note that the main evidence of lilov that increased my confidence that ivanov is quilty is not the good moves of ivanov but the fact that there are also games that he played like a weak player the way that he lost against weak players.

I agree that I saw no statistical proof based on matchs with houdini in the analysis of lilov.

The problem is some needed more then that, and the chess community needed more then that against BI. And FM Lilov gave it to them, and they accepted it like kids with candy.

The Lilov-Houdart testing protocol is pure BS. There is not a nice way to put it. It was accepted and touted because it seemed like good science, it sounded like good science. And the most imported thing was it did a hatchet job on BI. This is what Chessbase wanted, and this is what chess players wanted, and it is what FM Lilov wanted.

The Lilov-Houdart testing protocol was never tested. Accept by a few of us that were showing games with even higher computer move matches then BI. Which *they* said can't happen unless the player is cheating. Not my words, not my theory, not my testing protocol. But it sounds good and makes good print, but the Lilov-Houdart testing protocol was never tested to see if it was even remotely true.

Instead of testing to see if the protocol worked. Chessbase gave us Robert Houdart giving his blessing on what FM Lilov was doing. And confirming that no human can play like Houdini 3.

I don't believe in framing a guilty man, because I don't have hard evidence.

I don't want this to be the testing standard used against anyone for the reasons stated.

[Don]

I would like to run all these tournaments and ones that Borislav is not in. It's a matter of finding the PGN file for these tournaments. If anyone has any pointers to them I would like to grab them up. Otherwise I'll find them myself if they are out there. Also I would like to run the tournaments that included Borislav but where his performance was super low, like the ones where he performed less than 2100.

I just modified the test to hopefully improve it and I'm running it again. I am using Stockfish to test the moves for "easiness" so that we are not getting high match percentages on games full of forcing moves. If Stockfish would play some move from depth 1 to 12 and it's the same move and never changes, I consider it a trivial move. Nevertheless I will still test the move against Houdini's choice(s). It will still count as a match if Houdini plays a different move and so does the player being tested. Otherwise it's not counted as a data point.

A further refinement is that I won't match moves from the first 6 ply. The logic of this is that most basic tactics are resolved in the first few ply. Houdini can blunder on a 1 ply search tactically and so can a human (and for the same reason) but a cheater is never going to play one of these moves.

Now we are talking. Keep the data coming, and the methods used. I love good data.

You should sell this application it could have many uses.

If you can test other tournaments like Tal 2013. I would like to know what the top players percentage for comparison.

I was curious who Dimitrov, Radoslav was.

FIDE Chess Profile

2908549 Dimitrov, Radoslav
Federation Bulgaria
FIDE title International Master
Rating:
std.
2423 rapid
2438 blitz
2448

B-Year 1993
Sex Male

Don you may find this useful regarding BI tournaments and what to test.

This is one of the tournaments "Veliko Tarnovo BUL" FM Lilov claimed BI was cheating. BI scored 8.0/9.0.

[Mike S.]

The discussion has made huge progress. - Maybe, for convenience a new thread can be started, e.g. after you Don have a new and bigger analysis result.

Prof. Kenneth Regan, http://www.cse.buffalo.edu/~regan/ has done analyses on the topic of computer cheating in chess. He also paid attention to the B.I. case:

http://www.cse.buffalo.edu/~regan/chess ... curial.txt

What I find funny is that the figure of 73% appears again. (But in my analysis, it was just one Capablanca game and Houdini 1.5a.)

As for the Lilov method, if he really accepts any top-3 Houdini choices and even worse, from anytime throughout the calculation: Then, being a humble hobby player without elo, I would really be disappointed couldn't I score 67% engine matches. I mean, even I should be capable to at least find the worst of any top-3 Houdini choices, in 2 of 3 positions.

Regarding ChessBase, and in particular Chessbase.com I think that they are just reporting things and have no hidden agenda pro or contra Ivanov themselves. I am sure they are very open for reasonable opinions and analysis data on the matter. Their articles contain a contact link each, e.g.

http://chessbase.com/contact/MailToNews ... est-120713

The first major step would be to disprove Lilov's methodology with knowledgeable arguments, and preferably also with significant data examples (Capablanca, Fischer...).

As for paying attention to the surrounding circumstances and not just to the moves, I think this is indeed an important point. Regarding this, I am again pointing to IM Andrew Martin who has published YouTube videos about analysing Ivanov games, giving his expertise chess-wise.

http://www.youtube.com/user/YMChessMast ... ery=ivanov

Baseline: The moves in question are not beyond human grasp.

[Terry McCracken]

The discussion has made huge progress. - Maybe, for convenience a new thread can be started, e.g. after you Don have a new and bigger analysis result.

Prof. Kenneth Regan, http://www.cse.buffalo.edu/~regan/ has done analyses on the topic of computer cheating in chess. He also paid attention to the B.I. case:

http://www.cse.buffalo.edu/~regan/chess ... curial.txt

What I find funny is that the figure of 73% appears again. (But in my analysis, it was just one Capablanca game and Houdini 1.5a.)

As for the Lilov method, if he really accepts any top-3 Houdini choices and even worse, from anytime throughout the calculation: Then, being a humble hobby player without elo, I would really be disappointed couldn't I score 67% engine matches. I mean, even I should be capable to at least find the worst of any top-3 Houdini choices, in 2 of 3 positions.

Regarding ChessBase, and in particular Chessbase.com I think that they are just reporting things and have no hidden agenda pro or contra Ivanov themselves. I am sure they are very open for reasonable opinions and analysis data on the matter. Their articles contain a contact link each, e.g.

http://chessbase.com/contact/MailToNews ... est-120713

The first major step would be to disprove Lilov's methodology with knowledgeable arguments, and preferably also with significant data examples (Capablanca, Fischer...).

As for paying attention to the surrounding circumstances and not just to the moves, I think this is indeed an important point. Regarding this, I am again pointing to IM Andrew Martin who has published YouTube videos about analysing Ivanov games, giving his expertise chess-wise.

http://www.youtube.com/user/YMChessMast ... ery=ivanov

Baseline: The moves in question are not beyond human grasp.

This is all so idiotic. I'm going to find Ivanov and beat him senseless...

[Don]

I have most of the first tournament completed using the new criteria:

1. Start at move 15
2. ignore easy moves unless they do NOT match.
3. ignore Houdini moves below 7th iteration (don't match those.)

So basically if Houdini changes his mind any moves considered on 7th ply and beyond can be a match.

For a move to be considered easy or trivial Stockfish must like it on depth 1-12 and never switch away. If Houdini and the player like a different move it is not considered easy.

Here is a run of the single Carlsen game:

Code: Select all

match perc   matches   samples  games  player
----------  --------  --------  -----  -----------------
     66.67        18        27      1  Magnus Carlsen
     63.33        19        30      1  Teimour Radjabov

and here is the tournament run in progress:

Code: Select all

match perc   matches   samples  games  player
----------  --------  --------  -----  -----------------
    100.00         3         3      2  Lewicki, Pawel
     67.31        35        52      6  Ivanov, Borislav
     57.89        22        38      9  Galunov, Todor
     53.97        68       126      7  Nikolov, Sasho
     52.83        84       159      9  Dimitrov, Radoslav
     51.06        48        94      5  Nikolovski, Nikola
     50.00        68       136      9  Nevednichy, Boris M
     50.00         6        12      2  Stefanov, Evtim
     50.00         3         6      1  Todorov, Petar G
     48.15        13        27      2  Ilijkov, Ivan
     48.15        13        27      1  Vasilev, Martin
     48.15        13        27      1  Sisoev, Robert
     48.04        49       102      9  Rombaldoni, Axel
     47.37        45        95      7  Dimitrov, Dejan
     46.81        22        47      4  Kazakov, Peter
     46.53        47       101      6  Petrov, Vladimir Sergeev
     46.15        66       143      9  Erwich, Frank
     45.99        63       137      8  Maciol, Ryszard
     45.12        37        82      4  Ivanova, Simoneta
     44.68        42        94      6  Petkov, Emil
     44.63        54       121      8  Kalchev, Stefan
     44.44        48       108      3  Ljangov, Petar
     44.05        37        84      8  Janev, Pavel
     43.90        18        41      5  Petrov, Martin
     43.75         7        16      3  Iliev, Nikolay
     43.37        36        83      6  Angelov, Angel Y
     43.18        19        44      3  Stoyanov, Ivaylo
     42.98        49       114      9  Tashkov, Rumen
     42.86        21        49      2  Mitev, Miroslav
     42.48        48       113      6  Veselinov, Dimcho
     42.31        33        78      4  Atanasov, Anatoli
     42.05        37        88      4  Romcovici, Victor
     42.03        29        69      6  Ninov, Dayan
     41.84        82       196      8  Dimitrov, Pavel
     41.00        41       100      9  Katov, Luboslav
     40.71        46       113      8  Drenchev, Petar
     40.58        28        69      5  Galunova, Tsveta
     40.54        15        37      5  Dereshki, Dario
     40.51        32        79      9  Stanojoski, Zvonko
     40.43        19        47      4  Nikolov, Nikolay Petrov
     40.00        40       100      9  Stamenkov, Vanco
     40.00         4        10      1  Nikolovska, Dragana
     40.00         4        10      1  Nancheva, Doroteya
     39.74        60       151      8  Samu, Sorin-Mihai
     39.62        42       106      8  Kukov, Velislav
     39.56        72       182      9  Gazis, Efstathios
     39.29        11        28      1  Lazarov, Janaki
     39.06        25        64      3  Milea, Florian
     38.46        15        39      2  Sotirov, Ilia
     36.73        18        49      5  Stoyanov, Valeri
     36.36         4        11      1  Atanasov, Viktor Atanasov
     35.94        46       128      9  Ivelinov, Hristo
     35.57        53       149      9  Senetia, Teodor
     34.52        29        84      4  Kamenov, Valentin
     33.72        29        86      5  Shentov, Petar-Delian
     33.33         8        24      2  Mahlev, Atanas
     33.33         7        21      3  Stoyanov, Tihomir
     33.33         1         3      1  Simeonov, Svetoslav
     32.69        17        52      2  Lim, Kian Hwa
     31.58         6        19      1  Pencheva, Iva
     31.25        10        32      3  Stefanov, Emil
     31.03        18        58      7  Bochev, Krasimir
     30.77         8        26      9  Angelov, Kosta
     30.56        11        36      3  Marjanovics, Annamaria
     29.63         8        27      3  Marjanovics, Gyorgy
     28.57         2         7      1  Mihtis, Theodoros
     28.57         2         7      1  Ivanov, Yordan Kr
     23.26        10        43      3  Kolev, Velio
     21.28        10        47      2  Apostu, Toader
     20.00         9        45      6  Ilchev, Plamen
     16.00         8        50      6  Tzouvelekis, Ioannis
      0.00         0         6      1  Ivanov, Oleg
      0.00         0         4      1  Malinov, Boyan