Annotating a game using a database of annotated pgn games

[Vikram]

I had to do one change iteritems() to items() to make it work on my version of python

Its a very nice tool tool
Thanks

[giovanni]

CQL would be a reasonable solution to shortcut the number of games to feed to your script, though you would still get a bunch of useless positions at the first few moves. Also preselecting the games by ECO codes would certainly be useful, but probably still not enough in case of large reference database files...

Anyway, Ferdinand I really would like to thank you for all your efforts and the time you spend to help people on this board.

Giovanni

[Ferdy]

CQL would be a reasonable solution to shortcut the number of games to feed to your script, though you would still get a bunch of useless positions at the first few moves. Also preselecting the games by ECO codes would certainly be useful, but probably still not enough in case of large reference database files...

Anyway, Ferdinand I really would like to thank you for all your efforts and the time you spend to help people on this board.

Giovanni

I am using now pgn-extract to extract games from a given position, very fast. So there is no need for ECO position related searching.

With ref pgn of around 900, and limiting comments per position at 5, a game can be finished in around a minute. This depends on the comments of ref.pgn of course. Sample games from Bilbao using AnnotatedGames.pgn from Hiarcs as ref.pgn.

Code: Select all

[Event "9th Masters Final 2016"]
[Site "Bilbao ESP"]
[Date "2016.07.13"]
[Round "1.2"]
[White "Carlsen, Magnus"]
[Black "Nakamura, Hikaru"]
[Result "0-1"]
[WhiteTitle "GM"]
[BlackTitle "GM"]
[WhiteElo "2855"]
[BlackElo "2787"]
[ECO "B20"]
[Opening "Sicilian"]
[Variation "Keres variation (2.Ne2)"]
[WhiteFideId "1503014"]
[BlackFideId "2016192"]
[EventDate "2016.07.13"]

1.e4 {>>>[Matthew chose this move as he noticed in his preparation, that
Emmanuel didn't have a set answer to this.] -- Turner, Matthew J vs Berg, Emanuel, 1-0, ARCTIC CHESS CHALLENGE 2010. } {>>>In honour of Larsen, of course! -- McShane, Luke vs Mok Tze Meng, 1-0, Khanty Mansiysk Olympiad. } {>>>Magnus is now a confirmed 'anglophile' when it comes to his choice
of opening for White... -- Carlsen, Magnus vs Nakamura, Hikaru, 1-0, London Chess Classic 2nd. } {>>>"I'm old enough to play 1 d4! When I was a teenager, I told myself
I would start playing 1 d4 when I was thirty but it didn't happen. It has
now happened in my forties." (Nigel Short, in the commentary room) -- Short, Nigel vs Kosintseva, Tatiana, 1-0, Tradewise Gibraltar Chess Festival. } {>>>"This becoming a habit ... 1 d4 is a better move because your pawn
is protected" (Short) -- Short, Nigel vs Kosintseva, Nadezhda, 1-0, Tradewise Gibraltar Chess Festival. }  
1...c5 {>>>[The late Tony Miles would smile down at his former colleague.] -- Van Oosterom, C. vs Sadler, M., 0-1, ROC Nova College GpA. } {>>>And, to the delight of the crowd,
Gelfand sticks with his old favourite, the Najdorf, eschewing his frequent
modern preference, the Petroff. -- Mamedyarov, Shakhriyar vs Gelfand, Boris, 0-1, 3rd matchgame, Kazan. } {>>>Some pundits thought a switch to 1...c5 was on the cards but
Kasparov consents to another trip back to the 19th century in Doctor 'Who'
Short's time machine... -- Short, Nigel vs Kasparov, Garry, 0-1, Your Next Move Blitz. } {>>>No Dragon today! The Caro strikes me as a good choice against
the highly-aggressive and ever-dangerous Williams, and Gawain was actually
hoping for White's next move... -- Williams, Simon vs Jones, Gawain, 0-1, 4NCL. } {>>>A surprise for David - Levon usually only plays the Pirc Defence
in blitz games. -- Howell, David vs Aronian, Levon, 1/2-1/2, London Chess Classic. }  
2.Ne2 {>>>Usually I prefer to play more critically, but this
tournament convinced me that it is best to make your opponent uncomfortable
wherever possible rather than play the lines that are objectively best. -- Kilpatrick, C. vs Rotstein, A., 1-0, 2nd Livigno Open A. } {>>>Usually I prefer to play more critically, but this
tournament convinced me that it is best to make your opponent uncomfortable
wherever possible rather than play the lines that are objectively best. -- Kilpatrick, C. vs Rotstein, A., 1-0, 2nd Livigno Open A. } {>>>The so-called 'Big Clamp'. At least, anglophone
chessplayers call it that. GM Stuart Conquest used the term in one of the
Gibraltar commentaries to Judit Polgar and the world's top woman player
looked rather mystified by the name! Sergei Movsesian is a habitual
'clamper' and Luke McShane is also an occasional 'clampmeister'. One
difference between the Big Clamp and the Grand Prix Attack is that the
light-squared bishop cannot develop via the f1-a6 diagonal, of course. -- Jones, G. vs Slavin, A., 1-0, London Classic FIDE Open. } {>>>Younger English GMs Jones and McShane like to go
'off-piste' in the Sicilian but Short shows that elder statesman can play
the same game. -- Short, Nigel D vs Jones, G., 1/2-1/2, Bunratty Open. }  d6 3.Nbc3 a6 4.g3 g6 5.Bg2 Bg7 6.d4 cxd4 7.Nxd4 Nf6 8.O-O O-O 9.b3 Nc6 10.Nxc6 bxc6 11.Bb2 Qa5 12.Na4 Bg4 13.Qe1 Qh5 14.f3 Bh3 15.g4 Qh6 16.Rd1 g5 17.Bc1 Bxg2 18.Kxg2 Qg6 19.h4 gxh4 20.Qxh4 d5 21.g5 dxe4 22.f4 e6 23.c4 Rfd8 24.Rde1 Ne8 25.Nc5 Nd6 26.Qf2 f5 27.Bb2 Nf7 28.Bxg7 Kxg7 29.Qg3 Rd6 30.Rd1 Rad8 31.Rxd6 Rxd6 32.Qc3+ Kg8 33.Rf2 Qh5 34.Qh3 Qd1 35.Qe3 e5 36.Qg3 Rg6 37.Kh2 exf4 38.Qxf4 Qh5+ 39.Kg1 Qd1+ 40.Kh2 Qh5+ 41.Kg1 Nxg5 42.Qb8+ Kg7 43.Qe5+ Kh6 44.Qf4 Qd1+ 45.Kh2 Qd4 46.b4 Kg7 47.Qc7+ Kh8 48.Qc8+ Rg8 49.Qxf5 Nf3+ 50.Kh3 Qd6  0-1

Code: Select all

[Event "9th Masters Final 2016"]
[Site "Bilbao ESP"]
[Date "2016.07.14"]
[Round "2.2"]
[White "Wei, Yi"]
[Black "Carlsen, Magnus"]
[Result "0-1"]
[WhiteTitle "GM"]
[BlackTitle "GM"]
[WhiteElo "2696"]
[BlackElo "2855"]
[ECO "B06"]
[Opening "Robatsch (modern) defence"]
[WhiteFideId "8603405"]
[BlackFideId "1503014"]
[EventDate "2016.07.13"]

1.d4 {>>>[Matthew chose this move as he noticed in his preparation, that
Emmanuel didn't have a set answer to this.] -- Turner, Matthew J vs Berg, Emanuel, 1-0, ARCTIC CHESS CHALLENGE 2010. } {>>>In honour of Larsen, of course! -- McShane, Luke vs Mok Tze Meng, 1-0, Khanty Mansiysk Olympiad. } {>>>Magnus is now a confirmed 'anglophile' when it comes to his choice
of opening for White... -- Carlsen, Magnus vs Nakamura, Hikaru, 1-0, London Chess Classic 2nd. } {>>>"I'm old enough to play 1 d4! When I was a teenager, I told myself
I would start playing 1 d4 when I was thirty but it didn't happen. It has
now happened in my forties." (Nigel Short, in the commentary room) -- Short, Nigel vs Kosintseva, Tatiana, 1-0, Tradewise Gibraltar Chess Festival. } {>>>"This becoming a habit ... 1 d4 is a better move because your pawn
is protected" (Short) -- Short, Nigel vs Kosintseva, Nadezhda, 1-0, Tradewise Gibraltar Chess Festival. }  g6 2.e4 
2...Bg7 {>>>[Not really expected.] -- Adams, Michael vs Carlsen, Magnus, 1-0, 39th Olympiad Men. }  3.Nc3 d6 4.Be3 a6 5.f4 b5 6.Nf3 Nd7 7.e5 Bb7 8.Bd3 c5 9.Be4 Bxe4 10.Nxe4 Nh6 11.dxc5 dxe5 12.c6 Nf6 13.Qxd8+ Rxd8 14.Nxf6+ exf6 15.c7 Rc8 16.Bb6 Kd7 17.Ba5 Nf5 18.O-O-O+ Kc6 19.Rd8 Nd6 20.fxe5 fxe5 21.Rd1 Nc4 22.Bc3 Rxc7 23.b3 Ne3 24.Bxe5 Nxd1 25.Rd6+ Kb7 26.Bxg7 Rg8 27.Bd4 Nc3 28.Rb6+ Kc8 29.Be5 Rd8 30.Kb2 Nd5 31.Bxc7 Kxc7 32.Rxa6 Kb7 33.Ra3 Ne3 34.c4 bxc4 35.Ne5 Rd2+ 36.Kc3 Rc2+ 37.Kd4 Nf5+ 38.Kd5 c3 39.Kc5 Nd6 40.Nc6 Ne4+ 41.Kb5 Kc7 42.Ra7+ Kd6 43.Ra4 f5 44.Kb6 Rxg2 45.Rd4+ Ke6 46.a4 Ra2 47.a5 Rb2 48.Kc7 Nc5 49.Rc4 Na6+ 50.Kb6 Rxb3+ 51.Kxa6 Kd5 52.Rb4 c2 53.Ne7+ Kc5 54.Rxb3 c1=Q 55.Rb6 Qc4+ 56.Kb7 Qe4+ 57.Nc6 Qd5 58.Kc7 Qd6+ 59.Kb7 Qd7+ 60.Kb8 f4  0-1

It is fun to look at different comments coming from different games from old games. I will make the limit on number of comments an option to control the annotation.

After some more tests, I will upload the script.

[giovanni]

Nice improvement, Ferdinand!
I agree with you that it is going to be a fun tool!

[Ferdy]

I had to do one change iteritems() to items() to make it work on my version of python

Its a very nice tool tool
Thanks

Thanks for the info.

[Ferdy]

Nice improvement, Ferdinand!
I agree with you that it is going to be a fun tool!

Code: Select all

D. Program development log
version2
1. Optimize searching of position
2. Use pgn-extract to search games based on position
3. Added refpgnfile option
4. Added maxcomment option for the max number of comments per position
5. Added startmovenum option for starting move number where comment
will be appended
6. Added the Date tag value in the source of comment, after the Event tag value
7. Required pgn-extract-17-21.exe for fast extraction of games
based on position

http://www.mediafire.com/download/cr4t3 ... ent_v2.zip

You need pgn-extract here.
https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/

[giovanni]

Nice improvement, Ferdinand!
I agree with you that it is going to be a fun tool!

Code: Select all

D. Program development log
version2
1. Optimize searching of position
2. Use pgn-extract to search games based on position
3. Added refpgnfile option
4. Added maxcomment option for the max number of comments per position
5. Added startmovenum option for starting move number where comment
will be appended
6. Added the Date tag value in the source of comment, after the Event tag value
7. Required pgn-extract-17-21.exe for fast extraction of games
based on position

http://www.mediafire.com/download/cr4t3 ... ent_v2.zip

You need pgn-extract here.
https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/

Thank you! Now it is blazingly fast! In three hours it crunched 6 games vs. a database of about 11000 games. I noticed that stderr produced an output of over 60 Mbytes, mostly coming from pgn-extract. Perhaps, a less verbose output could boost speed a little bit and improve user experience.
All in all is a great tool! Thanks again.

[Ferdy]

Thank you! Now it is blazingly fast! In three hours it crunched 6 games vs. a database of about 11000 games. I noticed that stderr produced an output of over 60 Mbytes, mostly coming from pgn-extract. Perhaps, a less verbose output could boost speed a little bit and improve user experience.
All in all is a great tool! Thanks again.

I have already enabled the -s option when pgn-extract is run.

pgn-extract is strict in parsing notations. Try to run your ref pgn file in pgn-extract so that it will correct some of the things that it does not like. Backup your ref pgn.

If you have a log of warnings and errors, save and examine it and perhaps send it to the author of pgn-extract if such warning/error messages are still within the pgn standard, so that they will disable such warnings in future versions.

My annotatedGames.pgn used as my ref pgn does not produce any warning messages from pgn-extract.

Even the command to clean the pgn.

Code: Select all

pgn-extract -onew_file.pgn your_ref.pgn

would already correct those notations that according to pgn-extract is not within the pgn standard.

[giovanni]

Thank you! Now it is blazingly fast! In three hours it crunched 6 games vs. a database of about 11000 games. I noticed that stderr produced an output of over 60 Mbytes, mostly coming from pgn-extract. Perhaps, a less verbose output could boost speed a little bit and improve user experience.
All in all is a great tool! Thanks again.

I have already enabled the -s option when pgn-extract is run.

pgn-extract is strict in parsing notations. Try to run your ref pgn file in pgn-extract so that it will correct some of the things that it does not like. Backup your ref pgn.

If you have a log of warnings and errors, save and examine it and perhaps send it to the author of pgn-extract if such warning/error messages are still within the pgn standard, so that they will disable such warnings in future versions.

My annotatedGames.pgn used as my ref pgn does not produce any warning messages from pgn-extract.

Even the command to clean the pgn.

Code: Select all

pgn-extract -onew_file.pgn your_ref.pgn

would already correct those notations that according to pgn-extract is not within the pgn standard.

Thanks, Ferdinand. Your suggestion to clean the file with pgn-extract was excellent. Now your script runs at least 5 times faster! Needless to say, also the output is much terser.
Thanks again.
Giovanni

[giovanni]

Thank you! Now it is blazingly fast! In three hours it crunched 6 games vs. a database of about 11000 games. I noticed that stderr produced an output of over 60 Mbytes, mostly coming from pgn-extract. Perhaps, a less verbose output could boost speed a little bit and improve user experience.
All in all is a great tool! Thanks again.

I have already enabled the -s option when pgn-extract is run.

pgn-extract is strict in parsing notations. Try to run your ref pgn file in pgn-extract so that it will correct some of the things that it does not like. Backup your ref pgn.

If you have a log of warnings and errors, save and examine it and perhaps send it to the author of pgn-extract if such warning/error messages are still within the pgn standard, so that they will disable such warnings in future versions.

My annotatedGames.pgn used as my ref pgn does not produce any warning messages from pgn-extract.

Even the command to clean the pgn.

Code: Select all

pgn-extract -onew_file.pgn your_ref.pgn

would already correct those notations that according to pgn-extract is not within the pgn standard.

Thanks, Ferdinand. Your suggestion to clean the file with pgn-extract was excellent. Now your script runs at least 5 times faster! Needless to say, also the output is much terser.
Thanks again.
Giovanni

Hello, Ferdinand. Would it be possible for you also to generate the .exe file of your script?
Thanks again.
Giovanni