jja: convert CTG books to PolyGlot format (and more!)

[alpltl]

Could you take a look at this test case I created: https://gitlab.com/beejaganita/jja-tests/

I took a brief look, and did some brief testing. I'll do more tests at the weekend and report back. Thank you very much. So initially I selected the first 3 games from your D96.pgn and created a bin with polyglot and then exported that bin with latest jja, git version v0.5.0-58-g023065b:

Code: Select all

⇒  pex -s --selectonly 1:3 D96.pgn > D96-sample.pgn
⇒  polyglot make-book -min-game 1 -pgn D96-sample.pgn -bin D96-sample.bin
PolyGlot 2.0.4 by Fabien Letouzey.
inserting games ...
4 games.
346 entries.
filtering entries ...
250 entries.
sorting entries ...
saving entries ...
all done!
⇒  jja edit D96-sample.bin -o D96-sample-jja.pgn
Creating output PGN file...
Success creating output PGN file.
Generating a tree of all variations in the book...
Success generating a tree of all variations in the book.
Saving the opening tree into the given PGN file...
Saving done.

Now D96-sample.pgn have 3 games, ie 3 lines. D96-sample-jja.pgn have 7 lines. I have loaded both of these PGNs to Scid clipbase, turned on the tree window to inspect, and narrowed the problem down to one of the games which is this:

[pgn]
[Event "TCh-RUS Premier 2023"]
[Site "Sochi RUS"]
[Date "2023.05.10"]
[Round "9.2"]
[White "Lazavik, Denis"]
[Black "Paravyan, D"]
[Result "1/2-1/2"]
[WhiteTitle "GM"]
[BlackTitle "GM"]
[WhiteElo "2544"]
[BlackElo "2601"]
[ECO "D96"]
[Opening "Gruenfeld"]
[Variation "Russian variation"]
[PlyCount "155"]
[HashCode "ffbeedcb"]
[TotalPlyCount "155"]
[WhiteFideId "13515110"]
[BlackFideId "4194985"]

1. d4 Nf6 2. c4 g6 3. Nc3 d5 4. Nf3 Bg7 5. Qb3 dxc4 6. Qxc4 a6 7. Bf4 c6 8. e4 b5 9. Qb3 O-O 10. Be2 Qa5 11. Bd2 b4 12. Na4 Nxe4 13. Bxb4 Qc7 14. O-O a5 15. Ba3 Be6 16. Bc4 Bxc4 17. Qxc4 Nd6 18. Bxd6 Qxd6 19. Rac1 e6 20. Qe2 Nd7 21. Rc4 e5 22. Rd1 exd4 23. Nxd4 Rae8 24. Qf1 Ne5 25. Nxc6 Qxd1 26. Qxd1 Nxc4 27. g3 Nxb2 28. Nxb2 Bxb2 29. Qd2 Re6 30. Nxa5 Bf6 31. Nb3 Rd8 32. Qc2 Ra8 33. h4 h5 34. Kg2 Rea6 35. Nc1 Bg7 36. Qd2 Kh7 37. Qd5 R8a7 38. Qf3 Ra3 39. Qe4 Bh6 40. Nd3 Rxa2 41. Ne5 Bg7 42. Nf3 Bf6 43. Qe3 Kg7 44. Ng5 Bxg5 45. hxg5 Ra8 46. Kh3 Kg8 47. f4 R2a5 48. Qb6 Rd5 49. Qc6 Rad8 50. Qb6 R8d7 51. Qb8+ Kg7 52. Qb2+ Kh7 53. Qb8 Kg7 54. Qb2+ Rd4 55. Qc3 Kh7 56. Qc5 Kg8 57. Qc8+ Rd8 58. Qc6 R4d5 59. Qb6 R5d6 60. Qc7 Rd1 61. Qc6 Rf1 62. Kg2 Re1 63. f5 gxf5 64. g6 Rd2+ 65. Kh3 Rde2 66. gxf7+ Kxf7 67. Kh4 Re6 68. Qf3 Kg7 69. Qxf5 R1e5 70. Qd3 Re4+ 71. Kh3 h4 72. Qc3+ Kg6 73. Qd3 hxg3 74. Kxg3 Kg5 75. Qd2+ Re3+ 76. Kf2 Kg6 77. Qxe3 Rxe3 78. Kxe3 1/2-1/2
[/pgn]

Now jja creates 3 lines for this game alone which look like this:

1. Line 1

[pgn]
[Event "D96-1.bin"]
[Site "rosarote"]
[Date "2023.07.14"]
[Round "1"]
[White "?"]
[Black "?"]
[Result "*"]
[Annotator "jja v0.5.0"]

1. d4 Nf6 2. c4 g6 3. Nc3 d5 4. Nf3 Bg7 5. Qb3 dxc4 6. Qxc4 a6 7. Bf4 c6 8. e4 b5 9. Qb3 O-O 10. Be2 Qa5 11. Bd2 b4 12. Na4 Nxe4 13. Bxb4 Qc7 14. O-O a5 15. Ba3 Be6 16. Bc4 Bxc4 17. Qxc4 Nd6 18. Bxd6 Qxd6 19. Rac1 e6 20. Qe2 Nd7 21. Rc4 e5 22. Rd1 exd4 23. Nxd4 Rae8 24. Qf1 Ne5 25. Nxc6 Qxd1 26. Qxd1 Nxc4 27. g3 Nxb2 28. Nxb2 Bxb2 29. Qd2 Re6 30. Nxa5 Bf6 31. Nb3 Rd8 32. Qc2 Ra8 33. h4 h5 34. Kg2 Rea6 35. Nc1 Bg7 36. Qd2 Kh7 37. Qd5 R8a7 38. Qf3 Ra3 39. Qe4 Bh6 40. Nd3 Rxa2 41. Ne5 Bg7 42. Nf3 Bf6 43. Qe3 Kg7 44. Ng5 Bxg5 45. hxg5 Ra8 46. Kh3 Kg8 47. f4 R2a5 48. Qb6 Rd5 49. Qc6 Rad8 50. Qb6 R8d7 51. Qb8 Kg7 52. Qb2 Kh7 53. Qb8 *
[/pgn]

2. Line 2

[pgn]
[Event "D96-1.bin"]
[Site "rosarote"]
[Date "2023.07.14"]
[Round "2"]
[White "?"]
[Black "?"]
[Result "*"]
[Annotator "jja v0.5.0"]

1. d4 Nf6 2. c4 g6 3. Nc3 d5 4. Nf3 Bg7 5. Qb3 dxc4 6. Qxc4 a6 7. Bf4 c6 8. e4 b5 9. Qb3 O-O 10. Be2 Qa5 11. Bd2 b4 12. Na4 Nxe4 13. Bxb4 Qc7 14. O-O a5 15. Ba3 Be6 16. Bc4 Bxc4 17. Qxc4 Nd6 18. Bxd6 Qxd6 19. Rac1 e6 20. Qe2 Nd7 21. Rc4 e5 22. Rd1 exd4 23. Nxd4 Rae8 24. Qf1 Ne5 25. Nxc6 Qxd1 26. Qxd1 Nxc4 27. g3 Nxb2 28. Nxb2 Bxb2 29. Qd2 Re6 30. Nxa5 Bf6 31. Nb3 Rd8 32. Qc2 Ra8 33. h4 h5 34. Kg2 Rea6 35. Nc1 Bg7 36. Qd2 Kh7 37. Qd5 R8a7 38. Qf3 Ra3 39. Qe4 Bh6 40. Nd3 Rxa2 41. Ne5 Bg7 42. Nf3 Bf6 43. Qe3 Kg7 44. Ng5 Bxg5 45. hxg5 Ra8 46. Kh3 Kg8 47. f4 R2a5 48. Qb6 Rd5 49. Qc6 Rad8 50. Qb6 R8d7 51. Qb8 Kg7 52. Qb2 Rd4 53. Qc3 Kh7 54. Qc5 Kg8 55. Qc8 Rd8 56. Qc6 *
[/pgn]

3. Line 3

[pgn]
[Event "D96-1.bin"]
[Site "rosarote"]
[Date "2023.07.14"]
[Round "3"]
[White "?"]
[Black "?"]
[Result "*"]
[Annotator "jja v0.5.0"]

1. d4 Nf6 2. c4 g6 3. Nc3 d5 4. Nf3 Bg7 5. Qb3 dxc4 6. Qxc4 a6 7. Bf4 c6 8. e4 b5 9. Qb3 O-O 10. Be2 Qa5 11. Bd2 b4 12. Na4 Nxe4 13. Bxb4 Qc7 14. O-O a5 15. Ba3 Be6 16. Bc4 Bxc4 17. Qxc4 Nd6 18. Bxd6 Qxd6 19. Rac1 e6 20. Qe2 Nd7 21. Rc4 e5 22. Rd1 exd4 23. Nxd4 Rae8 24. Qf1 Ne5 25. Nxc6 Qxd1 26. Qxd1 Nxc4 27. g3 Nxb2 28. Nxb2 Bxb2 29. Qd2 Re6 30. Nxa5 Bf6 31. Nb3 Rd8 32. Qc2 Ra8 33. h4 h5 34. Kg2 Rea6 35. Nc1 Bg7 36. Qd2 Kh7 37. Qd5 R8a7 38. Qf3 Ra3 39. Qe4 Bh6 40. Nd3 Rxa2 41. Ne5 Bg7 42. Nf3 Bf6 43. Qe3 Kg7 44. Ng5 Bxg5 45. hxg5 Ra8 46. Kh3 Kg8 47. f4 R2a5 48. Qb6 Rd5 49. Qc6 Rad8 50. Qb6 R5d6 51. Qc7 Rd1 52. Qc6 Rf1 53. Kg2 Re1 54. f5 gxf5 55. g6 Rd2 56. Kh3 Rde2 57. gxf7 Kxf7 58. Kh4 Re6 59. Qf3 Kg7 60. Qxf5 R1e5 61. Qd3 Re4 62. Kh3 h4 63. Qc3 Kg6 64. Qd3 hxg3 65. Kxg3 Kg5 66. Qd2 Re3 67. Kf2 Kg6 68. Qxe3 Rxe3 69. Kxe3 *
[/pgn]

What you can quickly notice is the extraneous lines are caused by the repetitions that takes place on moves 53 and 56. It's an open question whether we want to include these repetitions as new lines in the PGN, or we skip them, or maybe we make it optional through a command-line parameter. My initial opinion is it makes sense to keep them because it provides a clearer, and broader look at the opening book, however as usual I am open to discussion, suggestions, and thoughts.

[alpltl]

In the following position the move 4...0-0 is missing.
rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4
In the bin book after converting the little ctg book I shared here.
There are 4 games with the move 4...0-0 in the original CTG book.
I used the latest version of jja.exe you shared here on Talkchess.com

This is odd, it works for me with latest jja git version v0.5.0-58-g023065b, are we talking about the same CTG book?

Code: Select all

alip@rosarote:~/org/books/ctg|⇒  jja edit Carlsen_white.ctg -o Carlsen_white.bin
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Searching for move entries in the CTG input file. This may take a while.
Found 132121 move entries in the CTG input file.
Saving 132121 entries from input book to the output book...
Success copying entries from the input book to the output book.
Saving done.    [132121/132121] 21,796.4452/s [=======================================================================================================================] [00:00:06] ETA 00:00:00
alip@rosarote:~/org/books/ctg|⇒  jja find -f "rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4" Carlsen_white.bin
+---+------+--------+-------+
| * | UCI  | Weight | Learn |
+---+------+--------+-------+
| 1 | e8g8 | 65520  | 0     |
+---+------+--------+-------+
| 2 | b7b6 | 49140  | 0     |
+---+------+--------+-------+
alip@rosarote:~/org/books/ctg|⇒  jja find -f "rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4" Carlsen_white.ctg
+---+------+-----+-----+------+------+---------+-------------+----------------+
| * | UCI  | NAG | Win | Draw | Loss | Average | Performance | Recommendation |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 1 | e8g8 |     | 2   | 0    | 2    | 2922.75 | 2704.50     | 0x0            |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 2 | b7b6 |     | 1   | 1    | 0    | 2865.00 | 2706.00     | 0x0            |
+---+------+-----+-----+------+------+---------+-------------+----------------+

[alpltl]

In the following position the move 4...0-0 is missing.
rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4
In the bin book after converting the little ctg book I shared here.
There are 4 games with the move 4...0-0 in the original CTG book.
I used the latest version of jja.exe you shared here on Talkchess.com

This is odd, it works for me with latest jja git version v0.5.0-58-g023065b, are we talking about the same CTG book?

Note also, you can use "--no-scale" to avoid weight scaling, so here:

Code: Select all

alip@rosarote:~/org/books/ctg|⇒  jja edit Carlsen_white.ctg -o Carlsen_white.bin --no-scale
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Searching for move entries in the CTG input file. This may take a while.
Found 132121 move entries in the CTG input file.
Saving 132121 entries from input book to the output book...
Success copying entries from the input book to the output book.
Saving done.    [132121/132121] 22,361.8724/s [=======================================================================================================================] [00:00:05] ETA 00:00:00
alip@rosarote:~/org/books/ctg|⇒  jja find -f "rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4" Carlsen_white.bin
+---+------+--------+-------+
| * | UCI  | Weight | Learn |
+---+------+--------+-------+
| 1 | e8g8 | 4      | 0     |
+---+------+--------+-------+
| 2 | b7b6 | 3      | 0     |
+---+------+--------+-------+

So the first move, kingside castle for black, has two wins and two losses, b6 has 1 win and 1 draw, using the same calculation algorithm as polyglot which is 2*wins + draws we get 4, and 3. Another bit of information I can share is, to rescale such a book after it's created, you can use the "--rescale" argument to jja edit which is currently only in the git version:

Code: Select all

alip@rosarote:~/org/books/ctg|⇒  jja edit Carlsen_white.bin --rescale -i # change some comment in the editor window so the hash check passes through...
Parsing user submitted CSV file...
Success parsing user submitted CSV file with 16 entries.
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Copying 82556 entries from input book to the output book.
Saving user submitted edits to the output file.
Rescaling user submitted move weights.
Success saving 16 moves submitted by the user to the output file.
Success copying entries from the input book to the output book.
Copying done.   [82556/82556] 32,488.582/s [==========================================================================================================================] [00:00:02] ETA 00:00:00
alip@rosarote:~/org/books/ctg|⇒  jja find -f "rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4" Carlsen_white.bin
+---+------+--------+-------+
| * | UCI  | Weight | Learn |
+---+------+--------+-------+
| 1 | e8g8 | 65520  | 0     |
+---+------+--------+-------+
| 2 | b7b6 | 49140  | 0     |
+---+------+--------+-------+

Now we end up with the original scaled weights I shared with you in the previous post. I had to use this for some books distributed by Scid, such as Elo2400.bin and gm2600.bin to come up with more realistic weights to be used in engine play.

[Jonathan003]

This is odd, it works for me with latest jja git version v0.5.0-58-g023065b, are we talking about the same CTG book?

Yes we talk about the same CTG book. I only shard one CTG book here, with my Google drive: Carlsen_white.ctg
I used jja v0.5.0-55-g21b3767
Where can I find the latest version of jja.exe, to use on my Windows 11 64 bit PC?
I use Scid to navigate the converted book in bin format. What I mean by the move is not there, is that I can not see the move when browsing the book in Scid. I noticed when I make the move 4...0-0 on the board, in Scid, I can see whites reply 5. a3
So I can conclude that the move 4...0-0 is included in the bin book but not visible when browsing the book in Scid.
I have tried the "--no-scale" argument and the move 4...0-0 is still not visible.
I would like the move 4...0-0 to be visible when browsing the book in Scid. So it would be the same experience as when browsing the original CTG book in Chessbase.

[alpltl]

This is odd, it works for me with latest jja git version v0.5.0-58-g023065b, are we talking about the same CTG book?

Yes we talk about the same CTG book. I only shard one CTG book here, with my Google drive: Carlsen_white.ctg
I used jja v0.5.0-55-g21b3767
Where can I find the latest version of jja.exe, to use on my Windows 11 64 bit PC?
I use Scid to navigate the converted book in bin format. What I mean by the move is not there, is that I can not see the move when browsing the book in Scid. I noticed when I make the move 4...0-0 on the board, in Scid, I can see whites reply 5. a3
So I can conclude that the move 4...0-0 is included in the bin book but not visible when browsing the book in Scid.
I have tried the "--no-scale" argument and the move 4...0-0 is still not visible.
I would like the move 4...0-0 to be visible when browsing the book in Scid. So it would be the same experience as when browsing the original CTG book in Chessbase.

Thanks for the detailed information. I think what we witness here is another bug wrt. castling encoding for Polyglot books. I am working on it, I'll post an update when I have a fix.

[alpltl]

This is odd, it works for me with latest jja git version v0.5.0-58-g023065b, are we talking about the same CTG book?

Yes we talk about the same CTG book. I only shard one CTG book here, with my Google drive: Carlsen_white.ctg
I used jja v0.5.0-55-g21b3767
Where can I find the latest version of jja.exe, to use on my Windows 11 64 bit PC?
I use Scid to navigate the converted book in bin format. What I mean by the move is not there, is that I can not see the move when browsing the book in Scid. I noticed when I make the move 4...0-0 on the board, in Scid, I can see whites reply 5. a3
So I can conclude that the move 4...0-0 is included in the bin book but not visible when browsing the book in Scid.
I have tried the "--no-scale" argument and the move 4...0-0 is still not visible.
I would like the move 4...0-0 to be visible when browsing the book in Scid. So it would be the same experience as when browsing the original CTG book in Chessbase.

Thanks for the detailed information. I think what we witness here is another bug wrt. castling encoding for Polyglot books. I am working on it, I'll post an update when I have a fix.

I am happy to inform you that I have a fix for the bug you have reported. This affects not just ctg but abk, and obk book conversions as well. We've spotted another bug where the encoding of castling moves were incorrect. This commit fixes it. Please download the respective git build for your system, linux-glibc, or windows, and check if this fixes the problem for you.

Note, it is recommended to reconvert books with this build to ensure castling moves are encoded correctly. If you try to query a position where castling is one of the moves in a book generated by a buggy version of jja, you'll now get a warning like this:

Code: Select all

⇒  jja find -f "rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4" ~/org/books/ctg/Carlsen_white.bin
Failed to convert entry `BookEntry { key: 7638067310961601074, mov: 3902, weight: 65520, learn: 0 }' to move in position `rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq -'.
+---+------+--------+-------+
| * | UCI  | Weight | Learn |
+---+------+--------+-------+
| 2 | b7b6 | 49140  | 0     |
+---+------+--------+-------+

[Jonathan003]

Thanks for the fix, I will try the new version later.
I know out of experience that pawn promotion is not supported by the obk book format.
I find out when getting errors when trying to convert obk to pgn with the old obk2pgn command line tool.
The problem is not with obk2pgn but with the obk book format that doesn't support pawn promotion.

[alpltl]

Thanks for the fix, I will try the new version later.

You are welcome. Please try it out and let me know how it goes.

I know out of experience that pawn promotion is not supported by the obk book format.
I find out when getting errors when trying to convert obk to pgn with the old obk2pgn command line tool.
The problem is not with obk2pgn but with the obk book format that doesn't support pawn promotion.

This is interesting knowledge. Thanks for sharing. I know for a fact that CTG does not support underpromotion however never heard this restriction about OBK format. Do you have any obk file that is known to fail with obk2pgn? Maybe you can try them out with jja and see if jja fails too? I wonder how jja handles such cases and am eager to fix any bugs on the path.

[alpltl]

Thanks for the fix, I will try the new version later.
I know out of experience that pawn promotion is not supported by the obk book format.
I find out when getting errors when trying to convert obk to pgn with the old obk2pgn command line tool.
The problem is not with obk2pgn but with the obk book format that doesn't support pawn promotion.

Also speaking of OBK, do you have any OBK files with text notes? In theory, jja should support these files however I have no such file so I could not test it.

[alpltl]

Could you take a look at this test case I created: https://gitlab.com/beejaganita/jja-tests/

I took a brief look, and did some brief testing. I'll do more tests at the weekend and report back. Thank you very much.

I am happy to inform you that this bug has been fixed with the commit 14da746. The problem was happening due to jja expecting full-legality for en-passant moves rather than pseudo-legality. With this commit, jja is fully compatible with polyglot. To create identical books you should pass the options "--min-pieces=2 --no-scale" to jja make:

Code: Select all

⇒  polyglot | head -n1
PolyGlot 2.0.4 by Fabien Letouzey.
⇒  jja --version
jja v0.5.0-93-gdb49350
⇒  polyglot make-book -pgn D96.pgn -bin D96-pg.bin
PolyGlot 2.0.4 by Fabien Letouzey.
inserting games ...
101 games.
7496 entries.
filtering entries ...
110 entries.
sorting entries ...
saving entries ...
all done!
alip@rosarote:~/src/jja-tests|main⚡
⇒  jja make --min-pieces=2 --no-scale D96.pgn -o D96-jja.bin
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Parsing the input PGN files using 4 threads with max ply 1024, filter_side:None...
Opening input PGN file `D96.pgn'...
Success reading 100 games, with 0 filtered out, in 1 input PGN files.
Compacting the temporary rocksdb database. This may take a while.
Parsing done.   [100/?] 1,749.3896/s [00:00:00]
Found 7350 unique positions in the input PGN files.
Filtering the positions based on min_games:3 min_score:0 preserve_null:false
Filtering done. [7350/7350] 754,614.1576/s [==========================================================================================================================] [00:00:00] ETA 00:00:00
Saving 110 filtered entries to the output book.
Saving done.    [110/110] 65,448.9201/s [=============================================================================================================================] [00:00:00] ETA 00:00:00
⇒  sha1sum D96*.bin
4da9b6e5c64b3991defc005dd0ec7dbd382aa95d  D96-jja.bin
4da9b6e5c64b3991defc005dd0ec7dbd382aa95d  D96-pg.bin

As you can see from the sha1 checksums, the books created by polyglot and jja are identical. Please downlad a git build of your choice, linux-glibc, linux-musl, or windows, and check if this fixes the issue for you. With this I think we have fixed the most severe bugs fixed in CTG->BIN and PGN->BIN conversions. I plan to release this version as 0.6.0 soon. I hereby want to thank the talkchess community for their immense support in identifying critical bugs. Without this forum and the chessprogramming wiki, jja would not have been possible.