What is a reasonable value for mean-square-error when training NN based on game outcomes?

[osvitashev]

I have worked my way up to giving neural networks a try.

Here's what i have so far:
-Downloaded a collection of games from chess.com
-Transformed these games into a collection of FENs mapped to game outcome
-Assigned a loss a score of 0.0, a draw 0.5 and a win a 1.0

Now i am trying to train a neural network to predict the game outcome based on a position.
I am in a proof-of-concept stage and for now the network only takes piece mobility as the input.

The evaluation neural network based entirely on mobility seems to converge to MSE of about 0.2. I am assuming this is pretty bad in absolute terms, but at this point the network is oblivious to a lot of concepts essential for chess - such as pawns advancing towards promotion rank increasing in value.

My question is: is there a rule of thumb for what value would be a reasonable mean-square-error for the network to converge to?
(Also: is mean-square-error what people use as the error function? Is there one that works better?)

[brianr]

The MSE value by itself is not helpful.
The MSE trend for your net and training approach is useful.
It typically becomes lower quickly and then asymptotically become more flat.

How the NN plays games is the only thing that really matters.
For NNs of the same size and architecture, very fast fixed node per move games works pretty well.
For more different nets, time per move games are needed and take much longer, of course.

Ordo is useful. I like to see at least 95% CFS and prefer 99%.

[op12no2]

Similarly, my first net was based on the existing WDL labels (no scores) for about 8M positions from two well known data files: quiet-labeled.epd and lichess-big3-resolved.epd. Training a simple 768*64*1 (non-perspective) net converged to a MSE loss of about 0.07.

If you then use that 'boot' net to generate new data from self play for example, the loss will be typically be lower when you retrain because you can interpolate between the scores and the WDL labels. In my case I averaged them and my loss was 0.023 over 114M positions. The loss would be lower as you tended towards using scores and higher nearer WDL labels (I assume).

I'm now using that second net to generate more data...

There are other ways to 'boot': a random network, existing HCE, PSTs and endgame tablebases for example.