Yeah.. the problem is modern PGN files can be so big.Guenther wrote:I am still extracting one of those gigantic lichess databases out of curiosity.
The one I have downloaded (2018/01) probably will be around 35GB! decompressed.
I'll just mention my new command line utility sc_filter_pgn.tcl (in 'scripts' in the source tree).
It is helpful for people wanting to filter out a certain position from numerous large pgn files. It does not address the problem with a single pgn file maxing out the game limit, but some pgn sources come in increments.# sc_filter_pgn
# Using several PGN files, copy games matching position <fen> to a database
#
# Usage: sc_filter_pgn <database> <fen> <pgn-files....>