OpenCV Chess bot: visual chess board recognition & automated chess play on any website

[maksimKorzh]

Hey what's up guys, Code Monkey King's here.
Before hating me for building a tool allowing chess cheating
please bear in mind that it would be incredibly difficult to use it for a non-programmer
because in order to make it work one needs to:
1. Setup linux (it should work on windows as well but not tested)
2. Install OpenCV
3. Install Python & dependencies
4. Build a custom piece images dataset for visual piece detection purposes
(because of relying on absolute board position and size)
5. Implement timing logic for playing with humans
(it currently tuned to played vs stockfish bot on lichess)
But even if all above steps are made with a success still it's a proof of concept implementation
focused on board image to FEN string conversion on the one hand and best move to screen coordinates
on the other. So yeah, if someone would manage to cheat with this program than at very least he'll learn
how to code and then probably would realize that cheating is a bad thing)

Now from the developer's perspective here's the list of implemented options you might be potentially interested in:
1. Object detection using OpenCV (detecting chess pieces' coordinates)
2. Converting board image to FEN string
3. Converting best move in UCI format into absolute coordinates

HERE'S THE VIDEO DEMONSTRATION OF HOW IT WORKS:


HERE'S THE SOURCE CODE:
https://github.com/maksimKorzh/opencv-chess-bot

HERE'S THE TUTORIAL SERIES:

[AdminX]

I am sure the people on Chess.com, and Lichess.net will be working on a way to detect it, hell maybe even Chessbase, most people do not want to play a cheater. 'Build a better mouse trap and I will find a smarter mouse'.

[maksimKorzh]

I am sure the people on Chess.com, and Lichess.net will be working on a way to detect it, hell maybe even Chessbase, most people do not want to play a cheater. 'Build a better mouse trap and I will find a smarter mouse'.

As far as I'm aware cheating detection works via postmortem game analysis no matter how one inputs the moves.
At least on lichess... So the game played with this bot would instantly be classified as "100% stockfish perfect match"
which would lead to instant ban. So I can't see a need to detect "how the moves getting input" when it's literally
enough just to match the player's moves against best engine's line to detect a cheater.

Also it doesn't always properly recognize pieces if say king in check is highlighted, so to fix this behavior
one needs either to turn off highlights on a website or implement a more sophisticated object detection logic)

What this script does is like as if a human is having an engine in a separate window and makes moves
on 2 boards... It's kind of the same but more automated and more elegant)

But really - it's not about cheating but rather about visual object detection and board image to FEN conversion.
I think this technology is worth of highlighting because it can be used for real chess board's position detection
to supersede DGT boards potentially - imagine the following:
1. Camera takes a snapshot of a real chess board and converts it's position to FEN
2. Adjusts board position on screen to match the position on a chess board
3. If one has a robotic hand then it's possible to move opponents pieces on a real chess board)
Obviously this was implemented before on a highest level.

[willmorton]

I think this technology is worth of highlighting because it can be used for real chess board's position detection
to supersede DGT boards potentially - imagine the following:
1. Camera takes a snapshot of a real chess board and converts it's position to FEN
2. Adjusts board position on screen to match the position on a chess board
3. If one has a robotic hand then it's possible to move opponents pieces on a real chess board)
Obviously this was implemented before on a highest level.

Nice project!
To supersede DGT boards I guess it should also handle the time control and detect move times

[IanO]

I think this technology is worth of highlighting because it can be used for real chess board's position detection
to supersede DGT boards potentially - imagine the following:
1. Camera takes a snapshot of a real chess board and converts it's position to FEN
2. Adjusts board position on screen to match the position on a chess board
3. If one has a robotic hand then it's possible to move opponents pieces on a real chess board)
Obviously this was implemented before on a highest level.

Nice project!
To supersede DGT boards I guess it should also handle the time control and detect move times

I looked into sponsoring a university capstone project along those lines (also suggesting OpenCV). It is quite a bit more difficult to do 3D piece recognition in a natural light setting compared to a flat image of a diagram on the computer screen. The sticking points are the wide variety of piece styles, piece rotation (esp. for the knight), lower contrast, shadows, and piece obstruction. After several unsuccessful prototypes, we were looking into multiple cameras and an unobtrusive bar code at the base of the pieces for more accurate and reliable recognition. (Quite similar in concept to the RFID used by the DGT boards, just detected optically instead of by the board itself.)

[ThomasMiller]

I think this technology is worth of highlighting because it can be used for real chess board's position detection
to supersede DGT boards potentially - imagine the following:
1. Camera takes a snapshot of a real chess board and converts it's position to FEN
2. Adjusts board position on screen to match the position on a chess board
3. If one has a robotic hand then it's possible to move opponents pieces on a real chess board)
Obviously this was implemented before on a highest level.

Nice project!
To supersede DGT boards I guess it should also handle the time control and detect move times

I looked into sponsoring a university capstone project along those lines (also suggesting OpenCV). It is quite a bit more difficult to do 3D piece recognition in a natural light setting compared to a flat image of a diagram on the computer screen. The sticking points are the wide variety of piece styles, piece rotation (esp. for the knight), lower contrast, shadows, and piece obstruction. After several unsuccessful prototypes, we were looking into multiple cameras and an unobtrusive bar code at the base of the pieces for more accurate and reliable recognition. (Quite similar in concept to the RFID used by the DGT boards, just detected optically instead of by the board itself.)

Is piece recognition needed at all? Knowing whether a square is occupied or not should be enough given the initial FEN. The only tricky moves would be castling and promotions.

[maksimKorzh]

The entire idea of this project is to turn board screenshot to fen. Without this the project doesn't make sense.