How to generate a lot of unique chess positions?

[luki446]

Hello,

It may be a stupid question but I cannot figure it out by myself, I'm trying to generate a lot of chess positions for neural network training and I don't know how to do it efficiently. Right now I'm playing a few thousand games between a few chess engines on very low time control through cutechess and extract FENs from PGN but it gave me like 300 000 unique positions after like 16 hours and I need probably millions of them, is there any faster way to do it? Randomizing board state does not look like a solution

[amanjpro]

Hello,

It may be a stupid question but I cannot figure it out by myself, I'm trying to generate a lot of chess positions for neural network training and I don't know how to do it efficiently. Right now I'm playing a few thousand games between a few chess engines on very low time control through cutechess and extract FENs from PGN but it gave me like 300 000 unique positions after like 16 hours and I need probably millions of them, is there any faster way to do it? Randomizing board state does not look like a solution

Most people generate unique positions like what you do, and 300,000 is already very good. Zahak's first NNUE version only had 40M positions, same for BitGenie, and they both were around 3000 engines

[Uri Blass]

Hello,

It may be a stupid question but I cannot figure it out by myself, I'm trying to generate a lot of chess positions for neural network training and I don't know how to do it efficiently. Right now I'm playing a few thousand games between a few chess engines on very low time control through cutechess and extract FENs from PGN but it gave me like 300 000 unique positions after like 16 hours and I need probably millions of them, is there any faster way to do it? Randomizing board state does not look like a solution

I think that downloading a pgn of millions of games and searching for unique positions is a faster solution relative to playing games between engines at low time control.

I also do not understand why randomizing board state is not a solution.
I think that you can also use unbalanced positions when it is clear who is winning for testing when opponents play the position with both sides when the side who mate in less moves win.

It is a pity that some NNUE programs play poorly in unbalanced positions probably because they do not care about the number of moves to mate in testing.

[chrisw]

Self-play games to completion at short depth. It’s probably better to control the game generation (and output) yourself. Python Chess is useful here. I’m experimenting with random opening moves for random count of halfmoves, but with some control over the degree of random move selection. Also with junking the randomised opening sequences that are way out of range.
Then you need cores. With say 100 cores and 10ms per position generated, that’s 10^4 positions a second, or 24x60x60x10^4 positions a day, 0.9B per day.

[jdart]

Arasan uses self-play games at fixed depth, with the first few moves randomized. It also does some extra randomization by occasionally replacing the fixed-depth search with a search to a randomly generated number of nodes, and its selfplay utility uses a hashhtable to weed out duplicate positions before output. I have a lot of machines/cores but the bulk of the positions were generated on a 32-core Threadripper box.

[dangi12012]

Hello,

It may be a stupid question but I cannot figure it out by myself, I'm trying to generate a lot of chess positions for neural network training and I don't know how to do it efficiently. Right now I'm playing a few thousand games between a few chess engines on very low time control through cutechess and extract FENs from PGN but it gave me like 300 000 unique positions after like 16 hours and I need probably millions of them, is there any faster way to do it? Randomizing board state does not look like a solution

What you are asking is probably not what you really want. You want representative positions for beginning / mid / endgame since a random position will probably yield a midgame 99% of the time.

You could always generate random bitboards (12x uint64_t) and check for legality. (but that goes into above problem)
Since you do training - you can get millions of CPU hours of evaluation and position data for free:
https://database.lichess.org/

These pgns have the eval tag - this thread http://www.talkchess.com/forum3/viewtop ... =7&t=78464 knows how to parse it fast
There you have position + eval for training.
Godspeed in training.