Borislav Ivanov: a Lilov's add-on

[mwyoung]

I'm doing my own study now. I'm writing a little app which will examine a pgn file of games and compute the match percentage by some criteria. I'll report my findings.

In your app, to let it most correctly correlate with FM Lilov's method of detecting chess cheats.

The app should:

1. Disregard know opening theory.

2. Use forward progression in looking at the moves in the game.

3. Use some kind of time limit when looking at the moves.

4. Count as a computer hit or a confirmed move match, on any move from the game that shows in the top 3 choices of the engine being used, at any time during the search for that position. In FM Lilov's video, FM Lilov was using Houdini 3.

This best describes the method used by FM Lilov in his videos.

[Don]

Why are you being so defensive, you said you have not read and watch the all the articles and videos.

I know the method, that's way I was being clear. Since in this thread some have used a similar method,the blunder check method. But this is not exactly the same method used by FM Lilov. I have always used and talked about the correct method used against BI.

Now if you are done with the diversion, please answer the question.

Then explain this game for us Don. This game has a higher move correlation then any BI game using the Lilov-Houdart method of detecting chess cheats. And is clearly played by a human player.

I watched part of the video and I see them showing the top 3 moves of Houdini. If that is their matching criteria I think it is absolutely horrible.

I'm doing some experiments and I want to test many different players to get a kind of baseline - to see what a reasonable match percentage might be.

I don't know if this will be the final method but what I'm doing is skipping the first 15 moves (30 ply) since I cannot know where the book begins and ends, so this is just an arbitrary number. Then I match all moves up to move 80 and stop - or 4 moves from the final move so that I am not trying to match simple mop-up routines.

I consider a move a match if Houdini 3 would have played the move on ANY ITERATION - at any point before I stop the search. The search is run as if the clock has 60 seconds and 1 second increment, so these are roughly 2 or 3 second searches. Even at this, it takes a while to run a game and a big tournament takes hours.

I did run this game you post against my test using the above criteria and I come up with these match percentages:

Code: Select all

match perc   matches   samples  games  player
----------  --------  --------  -----  -----------------
     77.55        38        49      1  Magnus Carlsen
     71.43        35        49      1  Teimour Radjabov

You obviously don't know much about statics or you wouldn't be so obsessed about running tests on single games - but let me give you some basics so that we can talk the same language. Basically, in statistics what really matters is having large samples. For example I think even you would realize that if I play one move that Houdini plays it has no significance. It might for example be the most obvious move on the board, or the move following QxQ where PxQ is the only response. That would be a 100% match but it would be meaningless.

So is a 1 game sample enough? No. In the tournament I am testing right now some players agreed to a draw before move 15 and I'm starting at move 15! So one game generates a sample of zero! In the Capablanca cherry picked example we also have a short game that consists of moves that almost anyone would play once the opening has been left - even a computer doing a 1 ply search. So clearly not all moves are created equal. Even if we got 10 or 20 interesting positions it's a very small sample so it's important to sample a lot of moves before drawing any conclusion based solely on move matching.

However, I would like to mention that some people are basing their opinions on move matching only and ignoring everything else. But the only ones doing this are the ones that seem to be on Ivanov's side.

[Event "Norway Chess Tournament"]
[Site "Sandnes NOR"]
[Date "2013.05.14"]
[EventDate "2013.05.07"]
[Round "5"]
[Result "1-0"]
[White "Magnus Carlsen"]
[Black "Teimour Radjabov"]
[ECO "D32"]
[WhiteElo "?"]
[BlackElo "?"]
[PlyCount "135"]

1.c4 c5 2.Nf3 Nc6 3.Nc3 Nf6 4.e3 e6 5.d4 d5 6.cxd5 exd5 7.Bb5
Bd6 8.O-O O-O 9.dxc5 Bxc5 10.b3 Bg4 11.Bb2 a6 12.Bxc6 bxc6
13.Rc1 Ba7 14.Ne2 Qd6 15.Be5 Qe7 16.Ned4 Bxf3 17.Nxf3 Rfc8
18.Qd3 a5 19.Bxf6 Qxf6 20.Rc2 Rd8 21.Rfc1 c5 22.e4 Qg6 23.Re1
dxe4 24.Qxe4 Qxe4 25.Rxe4 Rd1+ 26.Re1 Rxe1+ 27.Nxe1 Rd8 28.Kf1
a4 29.bxa4 Rd4 30.a5 Ra4 31.Rd2 Kf8 32.Nd3 f6 33.Nb2 Rxa5
34.Nc4 Ra4 35.Rc2 Ke7 36.Ke2 Ke6 37.Kd3 Kd5 38.a3 h5 39.h3 h4
40.Rc1 g6 41.Rc2 g5 42.Rc1 Ra6 43.Re1 Bb8 44.Re7 Bf4 45.Kc3 f5
46.Kb3 g4 47.a4 gxh3 48.gxh3 Rg6 49.a5 Rg1 50.a6 Rb1+ 51.Kc3
Rc1+ 52.Kd3 Rd1+ 53.Ke2 Ra1 54.Nb6+ Kd6 55.Rg7 Kc6 56.Rg6+ Kb5
57.Nd5 Be5 58.Rb6+ Kc4 59.Ne3+ Kc3 60.f4 Bd4 61.Nxf5 c4 62.Rc6
Rh1 63.Nd6 Rh2+ 64.Kf3 Kd3 65.Rxc4 Rxh3+ 66.Kg4 Rh1 67.Ra4 Bf2
68.Ra3+ 1-0

[Don]

I'm doing my own study now. I'm writing a little app which will examine a pgn file of games and compute the match percentage by some criteria. I'll report my findings.

In your app, to let it most correctly correlate with FM Lilov's method of detecting chess cheats.

The app should:

1. Disregard know opening theory.

2. Use forward progression in looking at the moves in the game.

3. Use some kind of time limit when looking at the moves.

4. Count as a computer hit or a confirmed move match, on any move from the game that shows in the top 3 choices of the engine being used, at any time during the search for that position. In FM Lilov's video, FM Lilov was using Houdini 3.

This best describes the method used by FM Lilov in his videos.

I don't want to even know what method he is using right now - I don't want it to bias my own thinking. I want it to be a general tool so without a huge database of opening theory it's not easy to ignore opening theory - so I have to take an educated guess. I want to be able to run my app against large database of games and tournaments so I cannot manually check each one against opening theory.

I don't know what "forward progression" is.

I am using a time control for looking at the moves - I tell the program that it has 60 seconds on the clock and 1 second increment.

The top 3 choices is really stupid. If the player blunders it might still be the second or third choice of Houdini's if only 1 move is playable. Also, the top 3 choices is guaranteed to produce very high match rates with good players. It's VERY common for there to only be 2 or 3 good moves in a position so this idea is seriously broken unless Lilov is just trying to rig the results. I want a statistically valid test, not a contrived one.

Don

[mwyoung]

I made random choices from short games, just to limit the required analysis time but have a good depth at the same time. Maybe the 2nd example is more representative. I'm not sure if a 73% match would even be suspicious, in any modern master game.

If you apply a threshold, the number of disagreements is of course smaller. For example, I know a rather old Carlsen-Anand game, where there is only one disagreement with Houdini 1.5a beyond 10 centipawns. Amazing precision. That analysis was "only" at depth 16 though.

OTOH, if the intention is to process huge masses of data, maybe an even smaller depth is suitable? I am not sure. I think I have read of older experiments of similar kind where a depth of 10 or 12 was used (I don't recall exactly) some years ago, and engines which are now old.

It gets a bit confusing when talking about this subject. Because people are using different ways to confirm a computer move match.

Some here are using blunder check, that is backward progression of the game with some kind of thresh hold in centipawns to confirm a computer move match.

This works I guess to some extent, but this will clearly get you a lower percentage of hits then the method used by FM Lilov, and confirmed by Robert Houdart as the way of detecting chess cheating, with no further evidence being required to proclaim someone guilty of chess cheating with a computer program.

FM Lilov was doing forward progression, and basically any move that shows in the top 3 choices of the engine, as compared to the game move for any length of time. This was confirmed as a computer move match by FM Lilov.

A move match of 73% would be rather high using your method. That is a move match in almost every 3 of 4 moves. This would most likely be a higher percentage using the Lilov-Houdart method of move matching. When any move that matches the first 3 top choices of the computer would be a confirmed move match.

[mwyoung]

I'm doing my own study now. I'm writing a little app which will examine a pgn file of games and compute the match percentage by some criteria. I'll report my findings.

In your app, to let it most correctly correlate with FM Lilov's method of detecting chess cheats.

The app should:

1. Disregard know opening theory.

2. Use forward progression in looking at the moves in the game.

3. Use some kind of time limit when looking at the moves.

4. Count as a computer hit or a confirmed move match, on any move from the game that shows in the top 3 choices of the engine being used, at any time during the search for that position. In FM Lilov's video, FM Lilov was using Houdini 3.

This best describes the method used by FM Lilov in his videos.

I don't want to even know what method he is using right now - I don't want it to bias my own thinking. I want it to be a general tool so without a huge database of opening theory it's not easy to ignore opening theory - so I have to take an educated guess. I want to be able to run my app against large database of games and tournaments so I cannot manually check each one against opening theory.

I don't know what "forward progression" is.

I am using a time control for looking at the moves - I tell the program that it has 60 seconds on the clock and 1 second increment.

The top 3 choices is really stupid. If the player blunders it might still be the second or third choice of Houdini's if only 1 move is playable. Also, the top 3 choices is guaranteed to produce very high match rates with good players. It's VERY common for there to only be 2 or 3 good moves in a position so this idea is seriously broken unless Lilov is just trying to rig the results. I want a statistically valid test, not a contrived one.

Don

"The top 3 choices is really stupid."

It is not my testing protocol or my assertion this is FM Lilov and Robert Houdart protocol, but this is what was used in confirming a computer move match against BI. And is also one of my problems with this method.

Forward progressing is when you start the analysis of the game from move one, instead of starting at the end of the game and doing backwards analysis. Backwards progression of game analysis is used by Chessbase GUI, so the program engine can load the analysis of the previous moves into the hash tables when looking at the current move.

If your are using the app to check for move matching you will want to use forward progressing in analyzing. Otherwise it can skew the results in regards to move matching, since the program will have analyzed the previous moves and has stored the analysis into the hash.

[Cubeman]

These Borislav Ivanov threads are getting ridiculous. As I read it, there are 2 sides that don't agree and no amount of debating is going to work.
I was on the fence on deciding if B.I was guilty, but now have decided that it is most likely that he is. The major reason is that he demanded to be given a fair "trail" by the BCF and wanted the opportunity to clear his name.
The BCF set up a test of some sort to investigate B.I and needed him to appear, even his coach wanted him to turn up. But at the last minute he decided not to show. Can't think of a good reason for BI not to show, even if he thought the test was rigged against him, or the BCF had it in for him (which I doubt) then it still would be wise to do the test. He could "fail" the test but could later protest his innocence and show the whole world how the BCF had it in for him and expose the test so all could see how hard it was for a GM terminator like himself.

[mwyoung]

You obviously don't know much about statics or you wouldn't be so obsessed about running tests on single games

Why do you insist on questioning my knowledge of statistics, and trying to school me.

I am the one arguing against this for this very reason.

And I am using their testing protocol as FM Lilov did in his video to show how absurd it really is and calling it proof.

I am not the one who is showing a game of chess, using the top 3 choices of the computer to move match a single game as FM Lilov did in his video. And then claim this is proof someone cheated in that game.

So go and school the right people, and be outraged at FM Lilov that this is how we claim someone is guilty.

For me this is not about if BI is guilty or not, but I don't want this kind of testing protocol to be used as absolute proof that someone is guilty of computer chess cheating.

You were not even aware of that much basic detail in this case. You did not even know how FM Lilov was claiming BI was guilty.

Yes we can agree, FM Lilov's testing protocol is stupid. So I am making some progress with you.

[mwyoung]

Code: Select all

match perc   matches   samples  games  player 
 ----------  --------  --------  -----  ----------------- 
      77.55        38        49      1  Magnus Carlsen 
      71.43        35        49      1  Teimour Radjabov

For the some game using the Lilov-Houdart testing protocol GM Carlsen scored 94.5%.

Remember the theory is no human can play a game of chess like a computer. This is not my assertion Don, and it is not my testing protocol.

In the testing protocol used by FM Lilov no expections were made of any kind except for opening theory when move matching.

No game length was ever stated.

No amount of games was every stated.

If you watch all the video's you will understand why FM Lilov did not want to use a fair set of bench marks. FM Lilov thought BI was trying to hide the use of a computer at some points in the games, or in other games that BI played. FM Lilov did not want to show anything in his videos that FM Lilov did not think showed BI guilt.

FM Lilov setup this testing protocol to prove BI guilt in hindsight.

FM Lilov could not set up fair standards, because in some games BI had a low percentage of computer move matches. That FM Lilov would have to explain away those move and games.

FM Lilov said in his videos as one explanation for BI not move matching. When BI move matched the computer BI was cheating, when BI did not move match the computer BI either chose not to cheat, or BI cheating device malfunctioned.



I am not proclaiming if BI is guilty or not.

My problem is the testing protocol used, with no research of any kind to show it works by catching known cheaters, and not claiming guilt of innocent chess players.

I would hope no one would want FM Lilov's testing standard to be used. Because it was set up for the one reason. To do a hatchet job on BI. That makes for a poor testing protocol.

I don't have a problem with using some kind of testing protocol to detect cheating, and used as a tool. But it should be shown to work, and not flag chess players who are not cheating.

[Don]

Code: Select all

match perc   matches   samples  games  player 
 ----------  --------  --------  -----  ----------------- 
      77.55        38        49      1  Magnus Carlsen 
      71.43        35        49      1  Teimour Radjabov

For the some game using the Lilov-Houdart testing protocol GM Carlsen scored 94.5%.

Remember the theory is no human can play a game of chess like a computer. This is not my assertion Don, and it is not my testing protocol.

And it's not mine either and I have written before that I have not looked at their data to check it for myself and now that I know how they are doing it I completely disagree with their methodology.

No matter what methodology is used, including mine, you cannot just run the test on one game or games of one person - you have to test games from many tournaments and individuals in order to establish what is normal and what is not.

I think we can all agree that a reasonable player is going to match the moves of Houdini some percentage of time - but nobody really knows for sure what is outside the range of credulity. But we can do some studies to figure this out.

I couldn't care less about what Lilov is saying by the way, I like to see things for myself. I think it's very clear he is cheating even without the move matching but if it is to be properly used as evidence it needs to be done right.

We sometimes talk about innocent people being accused and convicted but the flip side of the coin is the number of guilty people who go free because the prosecutors were sloppy or made mistakes which cast doubts on all the evidence.

In the testing protocol used by FM Lilov no expections were made of any kind except for opening theory when move matching.

No game length was ever stated.

No amount of games was every stated.

If you watch all the video's you will understand why FM Lilov did not want to use a fair set of bench marks. FM Lilov thought BI was trying to hide the use of a computer at some points in the games, or in other games that BI played. FM Lilov did not want to show anything in his videos that FM Lilov did not think showed BI guilt.

FM Lilov setup this testing protocol to prove BI guilt in hindsight.

FM Lilov could not set up fair standards, because in some games BI had a low percentage of computer move matches. That FM Lilov would have to explain away those move and games.

FM Lilov said in his videos as one explanation for BI not move matching. When BI move matched the computer BI was cheating, when BI did not move match the computer BI either chose not to cheat, or BI cheating device malfunctioned.

I am not proclaiming if BI is guilty or not.

My problem is the testing protocol used, with no research of any kind to show it works by catching known cheaters, and not claiming guilt of innocent chess players.

I would hope no one would want FM Lilov's testing standard to be used. Because it was set up for the one reason. To do a hatchet job on BI. That makes for a poor testing protocol.

I don't have a problem with using some kind of testing protocol to detect cheating, and used as a tool. But it should be shown to work, and not flag chess players who are not cheating.

I agree with you 100% here. I always considered the move matching stuff a kind of distraction - a technique typically used by the defense, pick a fight with something minor and try to make it the core of the matter when it isn't.

[Don]

I still have more games to run for this tournament, but here is what I have so far based on the games of the "Veliko Tarnovo BUL" - I'm not sure what tournament that is or how he did in that one. I sorted this by match percentage.

My table show how many moves matched there were out of how many samples and how many games were involved. I doubt my methodology is perfect but it's probably a whole lot better than tallying a match if it was one of several Houdini choices. I still cannot believe how silly that is.

It's not a large sample but at least we start to get a rough sense of what match percentage would be unusually high or low for any given player.

I'm pretty sure that even if you were getting advice from a computer and playing its first choice you would not get 100% match since you cannot be sure of the many variables involved, such as how much thinking time is allowed, the number of cores set and non-deterministic program behavior.

In the first game the move matching percentage was pretty low for Borislav, only 60% which is close to the median. The rest are quite high but I'm not showing individual games here. I will add that as a feature later to the script and I will test this again at a higher level of play for Houdini 3. I also would like to see how these same players do when matched against Stockfish or Komodo.

Code: Select all

match perc   matches   samples  games  player
----------  --------  --------  -----  -----------------
     69.61       126       181      6  Dimitrov, Radoslav
     69.09        38        55      4  Ivanov, Borislav
     68.15        92       135      4  Dimitrov, Dejan
     67.57        50        74      6  Stanojoski, Zvonko
     67.33        68       101      4  Nikolovski, Nikola
     66.67        74       111      6  Rombaldoni, Axel
     66.67        40        60      6  Galunov, Todor
     66.67         4         6      3  Petrov, Martin
     66.30        61        92      6  Katov, Luboslav
     65.47        91       139      5  Maciol, Ryszard
     65.00        39        60      2  Angelov, Angel Y
     64.38       103       160      6  Nevednichy, Boris M
     64.00        16        25      3  Iliev, Nikolay
     63.93        78       122      5  Nikolov, Sasho
     63.64        21        33      1  Vasilev, Martin
     63.41        52        82      4  Galunova, Tsveta
     62.81        76       121      6  Tashkov, Rumen
     62.73        69       110      4  Petkov, Emil
     62.71       111       177      5  Dimitrov, Pavel
     62.00        62       100      5  Drenchev, Petar
     61.48        83       135      6  Stamenkov, Vanco
     61.36        54        88      2  Ivanova, Simoneta
     61.29        38        62      3  Dereshki, Dario
     60.98        25        41      6  Angelov, Kosta
     60.00        18        30      1  Pencheva, Iva
     60.00         6        10      1  Stefanov, Evtim
     60.00         6        10      1  Mihtis, Theodoros
     59.76        98       164      6  Gazis, Efstathios
     59.49        47        79      5  Janev, Pavel
     59.21        45        76      3  Shentov, Petar-Delian
     58.97        23        39      1  Ilijkov, Ivan
     58.49        31        53      4  Ninov, Dayan
     58.39        87       149      6  Ivelinov, Hristo
     58.29       116       199      6  Erwich, Frank
     57.84        59       102      4  Veselinov, Dimcho
     57.58        38        66      2  Stoyanov, Ivaylo
     57.48        73       127      5  Senetia, Teodor
     57.14        24        42      1  Sisoev, Robert
     57.14         8        14      1  Nikolovska, Dragana
     56.86        29        51      3  Stefanov, Emil
     56.80        71       125      5  Kukov, Velislav
     56.10        23        41      2  Nikolov, Nikolay Petrov
     55.56        10        18      1  Kamenov, Valentin
     55.22        37        67      4  Petrov, Vladimir Sergeev
     55.00        33        60      3  Kazakov, Peter
     54.95        50        91      3  Milea, Florian
     54.43        86       158      5  Samu, Sorin-Mihai
     53.15        76       143      4  Kalchev, Stefan
     52.94         9        17      1  Nancheva, Doroteya
     52.63        20        38      2  Mahlev, Atanas
     52.63        10        19      1  Atanasov, Viktor Atanasov
     51.82        71       137      3  Ljangov, Petar
     51.47        35        68      2  Mitev, Miroslav
     51.22        21        41      2  Stoyanov, Valeri
     50.85        30        59      4  Bochev, Krasimir
     48.94        23        47      1  Lazarov, Janaki
     48.65        18        37      1  Lim, Kian Hwa
     48.48        16        33      3  Ilchev, Plamen
     48.11        51       106      3  Romcovici, Victor
     46.67        14        30      1  Sotirov, Ilia
     46.34        19        41      3  Marjanovics, Annamaria
     40.54        15        37      3  Marjanovics, Gyorgy
     38.89        14        36      1  Apostu, Toader
     33.33         3         9      1  Ivanov, Yordan Kr
     33.33         2         6      1  Malinov, Boyan
     33.33         1         3      1  Simeonov, Svetoslav
     30.00         3        10      1  Ivanov, Oleg
     28.95        11        38      3  Tzouvelekis, Ioannis
     28.57         2         7      1  Todorov, Petar G
     20.51         8        39      2  Kolev, Velio