Looking for high quality selection of Lichess games.

Discussion of anything and everything relating to chess playing software and machines.

Moderator: Ras

Jonathan003
Posts: 243
Joined: Fri Jul 06, 2018 4:23 pm
Full name: Jonathan Cremers

Looking for high quality selection of Lichess games.

Post by Jonathan003 »

I know I can download Lichess databases here: https://database.lichess.org/
But I'm looking for collections of high quality Lichess databases.
I already find a collection database of human vs human games between players of at least 2200 elo
https://rebel13.nl/download/data.html
I also find the lichess_elite_scid_2021-03 database
https://database.nikonoel.fr/
I'm looking for collections of high quality Lichess games. Like all standard human games played on Lichess.org, where one of the players have a minimum rating of 2000 Elo.
I want a database of at least 100,000,000 games
I know I could try to build this database myself from the databases I can find here: https://database.lichess.org/
But that will take a verry long time, and I don't know if it will even be possible on my PC, because I have limited hardware.
dkappe
Posts: 1632
Joined: Tue Aug 21, 2018 7:52 pm
Full name: Dietrich Kappe

Re: Looking for high quality selection of Lichess games.

Post by dkappe »

I’ve made use of lichess pgn’s, but my requirements were quite different. For filtering and processing I used a very modest PC with pgn-extract and python-chess. Unless you’re using a 386 with floppies, you should be able to use those tools to get what you want.
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
User avatar
jshriver
Posts: 1360
Joined: Wed Mar 08, 2006 9:41 pm
Location: Morgantown, WV, USA

Re: Looking for high quality selection of Lichess games.

Post by jshriver »

This could be done by using pgn-extract to dump only games of a certain ELO or higher. The only issue is that it will also include computer matches since there is no tag specifying human or computer.
Volker Pittlik
Posts: 628
Joined: Wed Mar 08, 2006 9:10 pm
Location: Murten / Morat, Switzerland
Full name: Volker Pittlik

Re: Looking for high quality selection of Lichess games.

Post by Volker Pittlik »

Jonathan003 wrote: Wed Feb 02, 2022 1:28 am ...
I'm looking for collections of high quality Lichess games. Like all standard human games played on Lichess.org, where one of the players have a minimum rating of 2000 Elo.
I want a database of at least 100,000,000 games
...
100,000,000? My private collections doesn't come even close to that number. And there s a lot commercial stuff inside. I guess such a number of high-quality games simply does not exits. The necessary size of the database depends on the purpose of it.

vp
Jonathan003
Posts: 243
Joined: Fri Jul 06, 2018 4:23 pm
Full name: Jonathan Cremers

Re: Looking for high quality selection of Lichess games.

Post by Jonathan003 »

Volker Pittlik wrote: Wed Feb 02, 2022 7:06 am
Jonathan003 wrote: Wed Feb 02, 2022 1:28 am ...
I'm looking for collections of high quality Lichess games. Like all standard human games played on Lichess.org, where one of the players have a minimum rating of 2000 Elo.
I want a database of at least 100,000,000 games
...
100,000,000? My private collections doesn't come even close to that number. And there s a lot commercial stuff inside. I guess such a number of high-quality games simply does not exits. The necessary size of the database depends on the purpose of it.

vp
With high quality games I mean games that are useful for me to study and to make tactics from. I'm rated about 1600 elo on chess.com
So games where one player has a rating of at least 2000 elo will do it for me.
Jonathan003
Posts: 243
Joined: Fri Jul 06, 2018 4:23 pm
Full name: Jonathan Cremers

Re: Looking for high quality selection of Lichess games.

Post by Jonathan003 »

dkappe wrote: Wed Feb 02, 2022 1:57 am I’ve made use of lichess pgn’s, but my requirements were quite different. For filtering and processing I used a very modest PC with pgn-extract and python-chess. Unless you’re using a 386 with floppies, you should be able to use those tools to get what you want.
I like PgnTools to remove duplicates and merge pgn databases the same time.
https://eman.zohosites.com/
I think its much faster than pgn-extract and you can see the progress, so you have some idea how long it is gone take.
whelanh
Posts: 6
Joined: Mon Jun 28, 2021 4:04 pm
Full name: Hugh Whelan

Re: Looking for high quality selection of Lichess games.

Post by whelanh »

In terms of high quality games, I feel time control is quite important. A large number of blitz and bullet games with many blunders won't help you much. Also lichess has a lot of games with very few moves (i.e. abandoned games). For my own filtering of lichess games I use pgn-extract with a time and move number (and ratings) filters to only retain games with at least 15 moves, time control of at least 1000 seconds, (and ratings of both players >= 2050). The pgn extract command to do that is:
.\pgn-extract.exe -t.\filterFile.txt -#10000 -bl15 .\lichessMonthlyDownload.pgn

where filterFile.txt is just a simple text file with three lines:
WhiteElo >= "2050"
BlackElo >= "2050"
TimeControl >= "1000"

The #10000 argument just separates the output into separate files of 10,000 games (I find this speeds up processing). This typically reduces the massive number of lichess games down to something like only 20,000 - 30,000 games per month.
Cornfed
Posts: 511
Joined: Sun Apr 26, 2020 11:40 pm
Full name: Brian D. Smith

Re: Looking for high quality selection of Lichess games.

Post by Cornfed »

Jonathan003 wrote: Wed Feb 02, 2022 12:44 pm
Volker Pittlik wrote: Wed Feb 02, 2022 7:06 am
Jonathan003 wrote: Wed Feb 02, 2022 1:28 am ...
I'm looking for collections of high quality Lichess games. Like all standard human games played on Lichess.org, where one of the players have a minimum rating of 2000 Elo.
I want a database of at least 100,000,000 games
...
100,000,000? My private collections doesn't come even close to that number. And there s a lot commercial stuff inside. I guess such a number of high-quality games simply does not exits. The necessary size of the database depends on the purpose of it.

vp
With high quality games I mean games that are useful for me to study and to make tactics from. I'm rated about 1600 elo on chess.com
So games where one player has a rating of at least 2000 elo will do it for me.
We all have our excentricities, but with that being your intention, you are truly wasting your time in trying to 'reinvent the wheel' here. Plenty of tactics trainers out there already exist to 'study'.
Jonathan003
Posts: 243
Joined: Fri Jul 06, 2018 4:23 pm
Full name: Jonathan Cremers

Re: Looking for high quality selection of Lichess games.

Post by Jonathan003 »

I think you don't understand the idea I try to explain. So you shouldn't judge.