Neural net for cheat detection

[dkappe]

The thing is ... Regan's model is practically useless for online cheat detection... I rejected it for good reasons and opted for an alternative. His model is more like a reference guide, and is the model used by FIDE.

Can you explain further (and why online and OTB would be different)?

[expositor]

If it worked, then cheaters could use such a network to avoid suspicious moves.

Although this is true for any method of cheat detection. Cheating has always been (and always will be, I think) an evolutionary arms race.

[scchess]

The thing is ... Regan's model is practically useless for online cheat detection... I rejected it for good reasons and opted for an alternative. His model is more like a reference guide, and is the model used by FIDE.

Can you explain further (and why online and OTB would be different)?

Sure. In OTB, all you have are the chess moves. Collecting engine scores from the moves are simple. Regan would then compare the metrics with a pre-computed population in order to generate a z-score for hypothesis testing. The model works fine on OTB because: 1). you don't really have other metrics 2.) often you can use the z-score in conjunction with physical evidence to confront the accuser

(2) is important because sometimes you'd suspect someone of cheating, but the evidence is not totally overwhelming like someone follow Stockfish's moves the whole game. Physical evidence such as arbitrators statement, playing hall etc is needed for the final decision.

His model totally melts down in online cheating where you don't have access to the playing environment. The uncertainly is much higher but the collected data is also much more. His model is unable to account for very high dimensional and sparse dataset often seen in online cheat detection. Also, in online cheating, it's not just about matching accuracy. For example, cheating 1-min bullet game with an automated mouse mover is common on lichess.

chess.com confirmed Regan's model was not used in their system, and so are lichess and PMG.

[dkappe]

Sure. In OTB, all you have are the chess moves. Collecting engine scores from the moves are simple. Regan would then compare the metrics with a pre-computed population in order to generate a z-score for hypothesis testing. The model works fine on OTB because: 1). you don't really have other metrics 2.) often you can use the z-score in conjunction with physical evidence to confront the accuser

(2) is important because sometimes you'd suspect someone of cheating, but the evidence is not totally overwhelming like someone follow Stockfish's moves the whole game. Physical evidence such as arbitrators statement, playing hall etc is needed for the final decision.

His model totally melts down in online cheating where you don't have access to the playing environment. The uncertainly is much higher but the collected data is also much more. His model is unable to account for very high dimensional and sparse dataset often seen in online cheat detection. Also, in online cheating, it's not just about matching accuracy. For example, cheating 1-min bullet game with an automated mouse mover is common on lichess.

chess.com confirmed Regan's model was not used in their system, and so are lichess and PMG.

Thanks. A rare bit of light instead of heat in these discussions.

I think Regan’s model was developed when the main mode of play was OTB. Your criticisms make sense. Its unfortunate (though understandable) that most major platforms feel the need to keep their cheat detection approach secret.

[scchess]

Sure. In OTB, all you have are the chess moves. Collecting engine scores from the moves are simple. Regan would then compare the metrics with a pre-computed population in order to generate a z-score for hypothesis testing. The model works fine on OTB because: 1). you don't really have other metrics 2.) often you can use the z-score in conjunction with physical evidence to confront the accuser

(2) is important because sometimes you'd suspect someone of cheating, but the evidence is not totally overwhelming like someone follow Stockfish's moves the whole game. Physical evidence such as arbitrators statement, playing hall etc is needed for the final decision.

His model totally melts down in online cheating where you don't have access to the playing environment. The uncertainly is much higher but the collected data is also much more. His model is unable to account for very high dimensional and sparse dataset often seen in online cheat detection. Also, in online cheating, it's not just about matching accuracy. For example, cheating 1-min bullet game with an automated mouse mover is common on lichess.

chess.com confirmed Regan's model was not used in their system, and so are lichess and PMG.

Thanks. A rare bit of light instead of heat in these discussions.

I think Regan’s model was developed when the main mode of play was OTB. Your criticisms make sense. Its unfortunate (though understandable) that most major platforms feel the need to keep their cheat detection approach secret.

Honestly, you don't need the platforms to publish their algorithms because ... cheat detection is just an anatomy application applying on the chess data. It's like catching credit-card frauds. Forming a statistical model for credit-card frauds is well documented. The book "Fraud Analysis" by Wiley covers it. I have the book and I enjoy it. It's the same thing but a different data domain.

One of the most challenges is data labelling. If we could label the games as "cheat vs non-cheat" perfectly, then we don't need to make a cheat detection system. But if we can't label them properly, how to construct a neural network supervised learning? This is a Chicken or the egg problem for cheat detection, and arguably the most challenging. This was my biggest challenge when during my cheat detection development. How to label someone as cheater? Can I do it manually? If so, how many games I can check? What would be the criteria and process? If I were to write a program to label the games, so why write a cheat detection system then? Can we label the game as multi-class problem (i.e. less likely, somewhat likely, very likely ...), if so how to determine the boundaries? Who are the people I can ask to help me for labelling? How to pay them? What false-positive-rate threshold to set for my cheat detection? How to do backtesting? What data to give to backtesting and make sure the data is clean? etc etc etc

Developing a cheat detection is much more than just a Kaggle-like competition, where you take the data and try to fit for the best test accuracy.

Unlike chess engine development, accuracy is not always the most important metric in cheat detection. Explainability is a key factor.

[dkappe]

There are a number of Wiley books with titles similar to that. Who is the author?

[scchess]

https://www.amazon.com.au/Analytics-Des ... 230&sr=8-1

[Fulvio]

Honestly, you don't need the platforms to publish their algorithms because ... cheat detection is just an anatomy application applying on the chess data. It's like catching credit-card frauds. Forming a statistical model for credit-card frauds is well documented. The book "Fraud Analysis" by Wiley covers it. I have the book and I enjoy it. It's the same thing but a different data domain.

I don't agree, there are things, like how much time it used to think on a move, that are specific to chess and can be used to detect cheating.

[scchess]

True, but they are just data features. This is what I said anatomy application applying on the chess data domain

[M ANSARI]

I think chess moves should not be the only metric used ... most likely there is a lot of information in statistical data of how a person performs over a period of time. It is impossible to check one game and tell if someone has cheated in that one game ... or even in one tournament. This is especially true with NNUE engines today ... they can give you a list of moves to choose from and you choose the most logical (for a human move). Even the strongest human player is about 1000 ELO points stronger ... add to that the possibility of using 7 piece EGTB's ... the gap is probably even larger. You could probably play many suboptimal moves that are not even on the radar of an engine ... and then still win. If you look at Hikaru's games against Komodo ... where Komodo gives him 1 pawn odds ... he couldn't win even 3 years or 4 years ago. So if you are a smart cheater you can obfuscate things by playing non computer moves where you now the game is still drawn or even losing (if very tactical position) and then start playing the strong engine moves. Really really difficult to flag this ... and the stronger the player the most difficult it is.

I don't know ... after watching Caruana's interview yesterday I feel that this is an uphill battle and am not too optimistic ... if this battle fails ... it is the end of competitive chess. Maybe there is a to prevent technically anyone from cheating ... I mean I am sure there are many ways to circumvent any of today's protocols if there is enough motivation. With prize money going up ... tournament operators better start investing heavily to stay ahead of the arms race!