Annotating a game using a database of annotated pgn games

[giovanni]

Hi everyone.
I am looking for a utility that would take as input a pgn game and a database of annotated pgn games and it would output the annotated game with the comments it finds for any matching position in the database of annotate games. It would be even better if you could annotate at once a multi-game pgn file.
Ideally the program should run from the command line of a linux system, but any other existing tool would be OK.
Thanks in advance.
Giovanni

[Ferdy]

Hi everyone.
I am looking for a utility that would take as input a pgn game and a database of annotated pgn games and it would output the annotated game with the comments it finds for any matching position in the database of annotate games. It would be even better if you could annotate at once a multi-game pgn file.
Ideally the program should run from the command line of a linux system, but any other existing tool would be OK.
Thanks in advance.
Giovanni

Can you give an example? It is not clear to me what is the output.

[giovanni]

Thanks, Ferdinand. I enclose an example of an annotated single game database to be used as input. Any time the program finds a matching position from the input database, it should copy the annotation (and possibly the source) from the database in the output.
So let's say that I have in the input file of a game that I want to annotate the moves: 1. c4 c5 2. Nc3 Nf6 3. g3 d5 4. cxd5 Nxd5 5. Bg2 Nc7. Program should find that in the annotated database there is a commented matching position and the comment is: "{The exchange in c3 and then g7-g6 was preferable.}". It should output the comment in the output, embedding it at the matching position, i.e., it should annotate the games based on the comments found in the input database.



[Event "003.Hastings (E.V)"]
[Site "?"]
[Date "1932.??.??"]
[Round "?"]
[White "Flohr, Salo"]
[Black "Rellstab, Ludwig"]
[Result "1-0"]
[ECO "A34"]
[Annotator "Pezzi/Carosi"]
[PlyCount "71"]
[EventDate "1932.??.??"]

1. c4 c5 2. Nc3 Nf6 3. g3 d5 4. cxd5 Nxd5 5. Bg2 Nc7 {
The exchange in c3 and then g7-g6 was preferable.} 6. b3 e5 7. Bb2 Be7 8. Rc1
O-O 9. Na4 Nd7 {A reply which is not unpleasant at all for White: the right
square for this Knight was c6. The c5 Pawn should be protected in a different
way.} 10. Nf3 f6 11. Qc2 Ne6 12. Nh4 Nb6 {Not 12... Nd4 for 13. Bxd4 cxd4 14.
Bd5+ Kh8 15. Qe4 with the threats 16.Bxb7 and, more important, 16. Ng6+ hxg6
17.Qh4+ mate.} 13. Nf5 {Not wasting time in the Knight exchange. Flohr wants
to have a clear edge in the middlegame and is not afraid of doubling the
a-file pawn.} 13... Nxa4 14. bxa4 Rb8 15. f4 exf4 16. gxf4 Re8 {Of course the
Pf4 is at present one of the "untouchables": 16... Nxf4 17. Qc4+ Ne6 18. Bd5
Qd7 19. Bxe6+ Qxe6 20. Nxe7+} 17. Rg1 {Flohr: "Up to now I was satisfied with
my play. But castling now would be stronger because, after moving the Rook to
f3, there would be more opportunities for an attack. I played this move
because I expected some defensive reply from my opponent. And he did not
disappoint me."} 17... Bf8 {Clearly not expecting White's reply. 17...Kh8 was
needed and Flohr should have protected the f4 Pawn with e2-e3 (not necessary
if he had castled at the 17th move).} 18. Bc6 {Here is the reason for 17.Rg1!.
Just a bit of imagination. Now the threat is 19.Nh6+ and 20.Nf7+.} 18... Nd4
19. Nh6+ {A mistake would be 19. Bxd4 for 19... Bxf5. 19. Nxd4 was also wrong
because of the continuation 19... bxc6 20. Nxc6 Rxe2+ 21. Kxe2 Qe8+ and Black
is better.} 19... Kh8 20. Nf7+ Kg8 21. Nh6+ Kh8 22. Nf7+ {Flohr: "I was in
severe zeitnot - only two minutes to the time control - and repeating moves
had helped me saving time. My opponent proposed a draw. I thanked him but
declined the proposal, because, in the mean time, I had found a good plan of
attack (I had calculated variations up to the 29th move)." My friends, here is
a "tough" chessplayer! Indeed he was, for sure! How many of us, with only two
minutes left to play to the 40th move or give checkmate would have declined
the proposal?} 22... Kg8 23. Qc4 Rxe2+ {
If 23... Be6 then 24. Nxd8 Bxc4 25. Bxe8 gains material.} 24. Qxe2 Nxe2 25.
Nxd8 Nxg1 {Perhaps 25... bxc6 was better but after 26.Kxe2 Ba6+ 27. Ke3 Rxd8
28. d4 the White Rook is in action also.} 26. Bd5+ Kh8 27. Nf7+ Kg8 28. Kf2
Nh3+ 29. Kg3 {so Knight's fate is decided already.} 29... b5 30. axb5 Bb7 31.
Bc4 a6 32. a4 axb5 33. axb5 Nxf4 34. Kxf4 h5 35. Rg1 Kh7 36. Bxf6 {Black resign
s. Flohr does not tell us how many seconds are left, but probably, at this
point he did not need so many of them.} 1-0

[Ferdy]

Thanks, Ferdinand. I enclose an example of an annotated single game database to be used as input. Any time the program finds a matching position from the input database, it should copy the annotation (and possibly the source) from the database in the output.
So let's say that I have in the input file of a game that I want to annotate the moves: 1. c4 c5 2. Nc3 Nf6 3. g3 d5 4. cxd5 Nxd5 5. Bg2 Nc7. Program should find that in the annotated database there is a commented matching position and the comment is: "{The exchange in c3 and then g7-g6 was preferable.}". It should output the comment in the output, embedding it at the matching position, i.e., it should annotate the games based on the comments found in the input database.

So we can have 2 files.
1. GamesToBeAnnotated.pgn
2. AnnotatedGamesAsReference.pgn

If (1) contains 4 games and (2) has 10 annotated games, all positions in every game in (1) will be checked against all positions in every game in (2). If the position is the same, extract the comments in (2) if there is, and add/append it as comments in the game in (1) that is currently being evaluated or parsed.

Check if my understanding is correct.

[giovanni]

That's perfect!

[Ferdy]

That's perfect!

I will try to make one, a script using python-chess lib.

Install python and also python-chess.

[giovanni]

That's perfect!

I will try to make one, a script using python-chess lib.

Install python and also python-chess.

I will. Actually I already installed them to run the other scripts you already developed.
Thanks again.

[Ferdy]

That's perfect!

I will try to make one, a script using python-chess lib.

Install python and also python-chess.

I will. Actually I already installed them to run the other scripts you already developed.
Thanks again.

I got a working version, sample testing. The source of the comment is --<wplayer> vs <bplayer>, <result>, <Event>

ref.pgn - The ref 2 annotated games.

Code: Select all

[Event "003.Hastings (E.V)"]
[Site "?"]
[Date "1932.??.??"]
[Round "?"]
[White "Flohr, Salo"]
[Black "Rellstab, Ludwig"]
[Result "1-0"]
[Annotator "Pezzi/Carosi"]
[ECO "A34"]
[EventDate "1932.??.??"]

1.c4 c5 2.Nc3 Nf6 3.g3 d5 4.cxd5 Nxd5 5.Bg2 Nc7
{ The exchange in c3 and then g7-g6 was preferable. } 6.b3 e5 7.Bb2 Be7 8.Rc1
O-O 9.Na4 Nd7
{ A reply which is not unpleasant at all for White: the right square for
this Knight was c6. The c5 Pawn should be protected in a different way. }
10.Nf3 f6 11.Qc2 Ne6 12.Nh4 Nb6
{ Not 12... Nd4 for 13. Bxd4 cxd4 14. Bd5+ Kh8 15. Qe4 with the threats
16.Bxb7 and, more important, 16. Ng6+ hxg6 17.Qh4+ mate. } 13.Nf5
{ Not wasting time in the Knight exchange. Flohr wants to have a clear edge
in the middlegame and is not afraid of doubling the a-file pawn. } 13...Nxa4
14.bxa4 Rb8 15.f4 exf4 16.gxf4 Re8
{ Of course the Pf4 is at present one of the "untouchables": 16... Nxf4 17.
Qc4+ Ne6 18. Bd5 Qd7 19. Bxe6+ Qxe6 20. Nxe7+ } 17.Rg1
{ Flohr: "Up to now I was satisfied with my play. But castling now would be
stronger because, after moving the Rook to f3, there would be more
opportunities for an attack. I played this move because I expected some defensive reply from my opponent. And he did not disappoint me." }
17...Bf8
{ Clearly not expecting White's reply. 17...Kh8 was needed and Flohr should
have protected the f4 Pawn with e2-e3 (not necessary if he had castled at
the 17th move). } 18.Bc6
{ Here is the reason for 17.Rg1!. Just a bit of imagination. Now the threat
is 19.Nh6+ and 20.Nf7+. } 18...Nd4 19.Nh6+
{ A mistake would be 19. Bxd4 for 19... Bxf5. 19. Nxd4 was also wrong
because of the continuation 19... bxc6 20. Nxc6 Rxe2+ 21. Kxe2 Qe8+ and
Black is better. } 19...Kh8 20.Nf7+ Kg8 21.Nh6+ Kh8 22.Nf7+
{ Flohr: "I was in severe zeitnot - only two minutes to the time control -
and repeating moves had helped me saving time. My opponent proposed a draw.
I thanked him but declined the proposal, because, in the mean time, I had found a good plan of attack (I had calculated variations up to the 29th
move)." My friends, here is a "tough" chessplayer! Indeed he was, for sure! How many of us, with only two minutes left to play to the 40th move or give checkmate would have declined the proposal? }
22...Kg8 23.Qc4 Rxe2+
{ If 23... Be6 then 24. Nxd8 Bxc4 25. Bxe8 gains material. } 24.Qxe2 Nxe2
25.Nxd8 Nxg1
{ Perhaps 25... bxc6 was better but after 26.Kxe2 Ba6+ 27. Ke3 Rxd8 28. d4
the White Rook is in action also. } 26.Bd5+ Kh8 27.Nf7+ Kg8 28.Kf2 Nh3+
29.Kg3 { so Knight's fate is decided already. } 29...b5 30.axb5 Bb7 31.Bc4 a6
32.a4 axb5 33.axb5 Nxf4 34.Kxf4 h5 35.Rg1 Kh7 36.Bxf6
{ Black resign s. Flohr does not tell us how many seconds are left, but
probably, at this point he did not need so many of them. } 1-0


[Event "World Ch 1990"]
[Site "Spain"]
[Date "1990.??.??"]
[Round "?"]
[White "Kasparov, Gary"]
[Black "Karpov, Anatoly"]
[Result "1-0"]
[ECO "A34"]

1.c4 c5 { Heading for English symmetrical. } 2.Nc3 Nf6 3.g3 d5 4.cxd5 Nxd5
5.Bg2 Nc7 { Keeping the knight. } 6.b3 e5 7.Bb2 Be7 8.Rc1 O-O 9.Na4 Nd7
{ Develop a piece before engaging white's forces. } 10.Nf3 f6 11.Qc2 Ne6
12.Nh4 Nb6 1-0

raw.pgn - The file to be annotated

Code: Select all

[Event "Linares"]
[Site "?"]
[Date "1990.??.??"]
[Round "?"]
[White "Kramnik"]
[Black "Anand"]
[Result "1-0"]
[ECO "A34"]
[EventDate "1990.??.??"]

1.c4 c5 2.Nc3 Nf6 3.g3 d5 4.cxd5 Nxd5 5.Bg2 Nc7 6.b3 e5 7.Bb2 Be7 8.Rc1 O-O
9.Na4 Nd7 10.Nf3 f6 11.Qc2 Ne6 12.Nh4 Nb6 13.Nf5 Nxa4 14.bxa4 Rb8 15.f4 exf4
16.gxf4 1-0

raw_out.pgn - The output file, the raw.pgn will not be touched.

Code: Select all

[Event "Linares"]
[Site "?"]
[Date "1990.??.??"]
[Round "?"]
[White "Kramnik"]
[Black "Anand"]
[Result "1-0"]
[ECO "A34"]
[EventDate "1990.??.??"]

1.c4 
1...c5 {Heading for English symmetrical. -- Kasparov, Gary vs Karpov, Anatoly, 1-0, World Ch 1990. }  2.Nc3 Nf6 3.g3 d5 4.cxd5 Nxd5 5.Bg2 
5...Nc7 {The exchange in c3 and then g7-g6 was preferable. -- Flohr, Salo vs Rellstab, Ludwig, 1-0, 003.Hastings (E.V). } {Keeping the knight. -- Kasparov, Gary vs Karpov, Anatoly, 1-0, World Ch 1990. }  6.b3 e5 7.Bb2 Be7 8.Rc1 O-O 9.Na4 
9...Nd7 {A reply which is not unpleasant at all for White: the right square for
this Knight was c6. The c5 Pawn should be protected in a different way. -- Flohr, Salo vs Rellstab, Ludwig, 1-0, 003.Hastings (E.V). } {Develop a piece before engaging white's forces. -- Kasparov, Gary vs Karpov, Anatoly, 1-0, World Ch 1990. }  10.Nf3 f6 11.Qc2 Ne6 12.Nh4 
12...Nb6 {Not 12... Nd4 for 13. Bxd4 cxd4 14. Bd5+ Kh8 15. Qe4 with the threats
16.Bxb7 and, more important, 16. Ng6+ hxg6 17.Qh4+ mate. -- Flohr, Salo vs Rellstab, Ludwig, 1-0, 003.Hastings (E.V). }  
13.Nf5 {Not wasting time in the Knight exchange. Flohr wants to have a clear edge
in the middlegame and is not afraid of doubling the a-file pawn. -- Flohr, Salo vs Rellstab, Ludwig, 1-0, 003.Hastings (E.V). }  Nxa4 14.bxa4 Rb8 15.f4 exf4 16.gxf4  1-0

Limitation:
* Only text comments are extracted from the ref.pgn. Replayable variations are not copied.
* Just run pgn-extract if you want to format the length of the game notation.

Check if this is enough.

If you have a ref database of say 5 annotated games, and sample raw.pgn, you may post it here and I will try to test it before releasing this script.

[giovanni]

Thank you, Ferdinand. Your script seems to work well and the limitations you point out are really marginal. I' dont have handy a small meaningful reference database, because I plan to feed your script with a large one, in order to extract comments from games. However, I'll gladly test your script with my large database and I'll let you know how it goes.
Thanks again.

[Ferdy]

Thank you, Ferdinand. Your script seems to work well and the limitations you point out are really marginal. I' dont have handy a small meaningful reference database, because I plan to feed your script with a large one, in order to extract comments from games. However, I'll gladly test your script with my large database and I'll let you know how it goes.
Thanks again.

All right, here it is.

http://www.mediafire.com/download/jpcyv ... comment.7z

I added GPLv3 license to comply with python-chess.
Note use ref.pgn as your source of annotated ref pgn file. There is bat file in it to show command line options.