Tool for pgn to fen with metadata?

[dangi12012]

Hello I am looking for a tool that can expand all the games in a lichess database to each and every FEN.
That would work exactly like this python tool
https://pypi.org/project/pgn2data/

But it strips the pgn of the %eval tags. Which are cloud computed evaluations for any position. (could be up to multiple SF14 cpu hours per position)
So does anyone have a tool that converts a pgn like this into a list which contains FEN + Evaluation?

Code: Select all

[Event "Rated Blitz game"]
[Site "https://lichess.org/ixbfCLJj"]
[White "sayiner"]
[Black "raika"]
[Result "1-0"]
[UTCDate "2016.06.30"]
[UTCTime "22:00:25"]
[WhiteElo "1940"]
[BlackElo "1927"]
[WhiteRatingDiff "+14"]
[BlackRatingDiff "-10"]
[ECO "A30"]
[Opening "English Opening: Symmetrical Variation"]
[TimeControl "120+3"]
[Termination "Normal"]

1. c4 { [%eval -0.04] } 1... c5 { [%eval 0.18] } 2. e3 { [%eval 0.11] } 2... d6 { [%eval 0.52] } 3. d4 { [%eval 0.48] } 3... cxd4 { [%eval 0.55] } 4. exd4 { [%eval 0.55] } 4... Nf6 { [%eval 0.46] } 5. Nf3 { [%eval 0.46] } 5... e6 { [%eval 0.5] } 6. Be2 { [%eval 0.34] } 6... Be7 { [%eval 0.37] } 7. d5 { [%eval 0.37] } 7... e5 { [%eval 0.36] } 8. O-O { [%eval 0.32] } 8... O-O { [%eval 0.23] } 9. Nc3 { [%eval 0.19] } 9... b6 { [%eval 0.61] } 10. Be3 { [%eval 0.59] } 10... Nbd7 { [%eval 0.55] } 11. Nd2 { [%eval 0.58] } 11... Bb7 { [%eval 0.58] } 12. b4 { [%eval 0.71] } 12... Ne8 { [%eval 0.61] } 13. Qc2 { [%eval 0.5] } 13... f5 { [%eval 0.45] } 14. f4 { [%eval 0.37] } 14... Rc8 { [%eval 0.62] } 15. Rad1 { [%eval 0.64] } 15... Bf6 { [%eval 1.06] } 16. Nb5 { [%eval 0.94] } 16... a6 { [%eval 0.89] } 17. Na7 { [%eval 0.99] } 17... Rc7?! { [%eval 1.86] } 18. Qxf5?! { [%eval 1.28] } 18... exf4 { [%eval 1.79] } 19. Rxf4 { [%eval 1.31] } 19... Bxd5?? { [%eval 5.57] } 20. Qxd5+ { [%eval 5.62] } 20... Kh8 { [%eval 5.51] } 21. Nc6 { [%eval 5.5] } 21... Qc8 { [%eval 5.68] } 22. Nd4 { [%eval 5.61] } 22... Ne5? { [%eval 7.52] } 23. Ne6 { [%eval 7.23] } 23... Nxc4? { [%eval 12.18] } 24. Nxc4 { [%eval 11.77] } 1-0

[Guenther]

Hello I am looking for a tool that can expand all the games in a lichess database to each and every FEN.
That would work exactly like this python tool
https://pypi.org/project/pgn2data/

But it strips the pgn of the %eval tags. Which are cloud computed evaluations for any position. (could be up to multiple SF14 cpu hours per position)
So does anyone have a tool that converts a pgn like this into a list which contains FEN + Evaluation?

...

A FEN cannot contain that information (eval) by definition.
What you look for is PGN => EPD (which can have opcodes like e.g. eval)
http://www.saremba.de/chessgml/standard ... tm#c16.2.5
Ad hoc I don't remember any tool that does this.

Well it looks like this might do sth like that?
https://rebel13.nl/misc/epd.html

[Deberger]

So does anyone have a tool that converts a pgn like this into a list which contains FEN + Evaluation?

Here ya go:

Code: Select all

cat sample.pgn | pgn-extract --fencomments --commentlines | awk '/eval/ {getline f; print f $0}' 

{ rnbqkbnr/pppppppp/8/8/2P5/8/PP1PPPPP/RNBQKBNR b KQkq c3 0 1 }{ [%eval -0.04] }
{ rnbqkbnr/pp1ppppp/8/2p5/2P5/8/PP1PPPPP/RNBQKBNR w KQkq c6 0 2 }{ [%eval 0.18] }
{ rnbqkbnr/pp1ppppp/8/2p5/2P5/4P3/PP1P1PPP/RNBQKBNR b KQkq - 0 2 }{ [%eval 0.11] }
{ rnbqkbnr/pp2pppp/3p4/2p5/2P5/4P3/PP1P1PPP/RNBQKBNR w KQkq - 0 3 }{ [%eval 0.52] }
{ rnbqkbnr/pp2pppp/3p4/2p5/2PP4/4P3/PP3PPP/RNBQKBNR b KQkq d3 0 3 }{ [%eval 0.48] }

etc..

How's that?

[dangi12012]

So does anyone have a tool that converts a pgn like this into a list which contains FEN + Evaluation?

Here ya go:

Code: Select all

cat sample.pgn | pgn-extract --fencomments --commentlines | awk '/eval/ {getline f; print f $0}' 

{ rnbqkbnr/pppppppp/8/8/2P5/8/PP1PPPPP/RNBQKBNR b KQkq c3 0 1 }{ [%eval -0.04] }
{ rnbqkbnr/pp1ppppp/8/2p5/2P5/8/PP1PPPPP/RNBQKBNR w KQkq c6 0 2 }{ [%eval 0.18] }
{ rnbqkbnr/pp1ppppp/8/2p5/2P5/4P3/PP1P1PPP/RNBQKBNR b KQkq - 0 2 }{ [%eval 0.11] }
{ rnbqkbnr/pp2pppp/3p4/2p5/2P5/4P3/PP1P1PPP/RNBQKBNR w KQkq - 0 3 }{ [%eval 0.52] }
{ rnbqkbnr/pp2pppp/3p4/2p5/2PP4/4P3/PP3PPP/RNBQKBNR b KQkq d3 0 3 }{ [%eval 0.48] }

etc..

How's that?

That is perfect my friend. To get literally millions of cpu hours of free evaluations do these steps:
1) Download - https://database.lichess.org/standard/l ... 08.pgn.bz2
2) Extract with multiple Threads - "pbzip2 -v -d -k -m1000 lichess_db_standard_rated_2021-08.pgn"
3) Convert to Position with Eval - "cat lichess_db_standard_rated_2021-08.pgn | pgn-extract --fencomments --commentlines | awk '/eval/ {getline f; print f $0}' > eval_2021-08.txt"

4) Profit