Maia Chess, Engine Similarity, and Detecting Clones

[Madeleine Birchfield]

The recent Maia Chess recently came out announcing that they trained neural networks to play like human players. They also announced on the website that they are also trying to train neural networks to predict the moves of individual players and play like individual players, and are succeeding at it:

In current work, we are pushing the modeling of human play to the next level: can we predict the moves a particular human player would make? It turns out that personalising Maia gives us our biggest performance gains. We achieve these results by fine-tuning Maia: starting with a base Maia, say Maia 1900, we update the model by continuously training on an individual player's gains. The plot shows that personalized Maias achieve 75% accuracy at predicting particular players' moves.

The preprint for this work is available as a preprint here.

In the future somebody will try to train Maia on, for example, Stockfish, and be able to create a Maia personalised to Stockfish with 75% similarity, which according to Ed Schroeder's SIMEX would act like a Stockfish clone on the level of Houdini vs Stockfish 8 and Fire vs Stockfish 7, yet have no copied code between the two codebases whatsoever. In this case, how in the world is anybody able to detect that a closed source engine like Houdini or Fire are suspicious and likely Stockfish clones (or clones of whatever other engine) in the definition that the open source engine community uses it (copied codebase), rather than simply a Maia-esque neural network model trained on Stockfish (or whatever other engine)? Should these Maia neural network models be also considered clones of the original engines by the computer chess community?

[Nay Lin Tun]

Leela and Stockfish move choices are at least 75-80% similar in TCEC .

If they are getting stronger, there will be more similarities in the future. ( You can see in TCEC that they frequently share same PV lines for 10 moves + or 20 ply +).

It is harder to predict moves in low rated players because of blunders left and right. ( The aim of maia is to make a human like blunder, not a random blunder from traditional engines)

[Madeleine Birchfield]

Leela and Stockfish move choices are at least 75-80% similar in TCEC .

If they are getting stronger, there will be more similarities in the future. ( You can see in TCEC that they frequently share same PV lines for 10 moves + or 20 ply +).

It might be true that latest Leela and Stockfish dev versions are at least 75-80% similar in TCEC due to convergence at long time controls, but that is not true about Leela and weaker engines like Stockfish 7 or 8, (or Ethereal or Komodo for that matter). But what Maia allows people to do is to train on Stockfish 7 or 8, (or Ethereal or Komodo), and predict their moves in a way that is completely independent from engine strength or time control at an accuracy rating of 75%.

It is harder to predict moves in low rated players because of blunders left and right. ( The aim of maia is to make a human like blunder, not a random blunder from traditional engines)

While that is the aim of the current people behind Maia, since Maia is an open-source project according to their website somebody could easily fork the Maia codebase, fiddle around with it, and use it for other uses, including using it to try to replicate traditional engine play. The reverse could also happen, nefarious people who clone open source engines for a living to sell to unsuspecting customers can now claim to use a Maia like neural network in the engine to explain any similarity away between their engine and Stockfish 7 or 8 (or whatever other engine), when in reality they did violate the GPL license and copy the code.

[AdminX]

Even with Tablebase, Leela plays some silly endgames, how is Maia going to tackle that problem at higher levels I wonder.

[carldaman]

Leela and Stockfish move choices are at least 75-80% similar in TCEC .

If they are getting stronger, there will be more similarities in the future. ( You can see in TCEC that they frequently share same PV lines for 10 moves + or 20 ply +).

It might be true that latest Leela and Stockfish dev versions are at least 75-80% similar in TCEC due to convergence at long time controls, but that is not true about Leela and weaker engines like Stockfish 7 or 8, (or Ethereal or Komodo for that matter). But what Maia allows people to do is to train on Stockfish 7 or 8, (or Ethereal or Komodo), and predict their moves in a way that is completely independent from engine strength or time control at an accuracy rating of 75%.

It is harder to predict moves in low rated players because of blunders left and right. ( The aim of maia is to make a human like blunder, not a random blunder from traditional engines)

While that is the aim of the current people behind Maia, since Maia is an open-source project according to their website somebody could easily fork the Maia codebase, fiddle around with it, and use it for other uses, including using it to try to replicate traditional engine play. The reverse could also happen, nefarious people who clone open source engines for a living to sell to unsuspecting customers can now claim to use a Maia like neural network in the engine to explain any similarity away between their engine and Stockfish 7 or 8 (or whatever other engine), when in reality they did violate the GPL license and copy the code.

75% may match SF, but then it depends on what the other 25% are doing. If there are blunders sprinkled in among the 25%, the 75% won't matter much in engine chess, where 'the chain is only as strong as its weakest link'.

[Alayan]

Maia is based on Leela.

This is fundamentally a way to produce a weaker Leela network that are good at predicting the moves of a weaker chess playing entity.

I don't think this is an effective approach at approximating a strong classical engine.

[corres]

The recent Maia Chess recently came out announcing that they trained neural networks to play like human players. They also announced on the website that they are also trying to train neural networks to predict the moves of individual players and play like individual players, and are succeeding at it:

In current work, we are pushing the modeling of human play to the next level: can we predict the moves a particular human player would make? It turns out that personalising Maia gives us our biggest performance gains. We achieve these results by fine-tuning Maia: starting with a base Maia, say Maia 1900, we update the model by continuously training on an individual player's gains. The plot shows that personalized Maias achieve 75% accuracy at predicting particular players' moves.

The preprint for this work is available as a preprint here.

In the future somebody will try to train Maia on, for example, Stockfish, and be able to create a Maia personalised to Stockfish with 75% similarity, which according to Ed Schroeder's SIMEX would act like a Stockfish clone on the level of Houdini vs Stockfish 8 and Fire vs Stockfish 7, yet have no copied code between the two codebases whatsoever. In this case, how in the world is anybody able to detect that a closed source engine like Houdini or Fire are suspicious and likely Stockfish clones (or clones of whatever other engine) in the definition that the open source engine community uses it (copied codebase), rather than simply a Maia-esque neural network model trained on Stockfish (or whatever other engine)? Should these Maia neural network models be also considered clones of the original engines by the computer chess community?

I "like" those system what forcing human participants playing weak chess.
It would be more correct if there would be no any restriction to use helps. So the "centaur chess" is a correct solving for playing online chess, and not the "pol(?) correct chess". In the present days using of electronic helpers is a very natural thing in every field of the life. Being a "Don Quiote" against modern life is not a lucky thing.

[Uri Blass]

Leela and Stockfish move choices are at least 75-80% similar in TCEC .

If they are getting stronger, there will be more similarities in the future. ( You can see in TCEC that they frequently share same PV lines for 10 moves + or 20 ply +).

It is harder to predict moves in low rated players because of blunders left and right. ( The aim of maia is to make a human like blunder, not a random blunder from traditional engines)

It is impossible to predict 70-80% of part of the players who are weak enough.

Try to predict even 50% of the moves of the random mover.
You have no chance.

[M ANSARI]

I have been playing a lot of online bullet chess and taking on Level 6 and Level 7 on Lichess. I add 2 sec per move and find it quite entertaining. The level given for the engine on Level 6 is 2300 ... but to be honest I feel it is much more difficult to beat at that time control than it is to beat someone who is 2500 to 2600. The thing is that it plays like a 3400 player most times and then overlooks a simple move that puts it in a totally lost endgame. I don't know how the algo works but it certainly not human like. What I found very interesting chesswise is that it seems to purposely blunder a pawn sometimes, but with the fast time control that "blunder" quickly turns to have devastating tactical counter play. This really has given me many ideas and I feel I respect the value of pawn sacrifices for dynamic play much more. To make an engine play like a human is extremely difficult as the strength of engines is tactical and that happens where humans are weakest.