jja: convert CTG books to PolyGlot format (and more!)

[alpltl]

I want to announce jja, a command line utility to interact with various chess file formats. It is still in its very early stages of development. The initial intention of the author was to convert their opening books which were saved with ChessBase's proprietary CTG format to the free and open PolyGlot format. Overtime they intend to add support for other chess file formats (cbh, abk, epd, pgn, si4, si5 and so on).

home: https://crates.io/crates/jja
git: https://git.sr.ht/~alip/jja

Installation is as simple as "cargo install jja" on Linux.
This is free software, licensed under GPL-3.0-or-later.

I hope you find it useful, please let me know what you think.

[alpltl]

As of version 0.3.0, jja supports reading/querying:

PolyGlot, aka bin
Arena, aka abk
ChessBase, aka ctg
ChessMaster, aka obk (version 1 and 2, w\o,with text notes)

opening book files, wheras it supports writing/converting to:

PolyGlot, aka bin

opening book files.

During opening book conversion, jja uses the information provided in various input opening book formats to come up with a move weight which accompanies the move in the PolyGlot opening file. jja also writes some custom numbers in the learn field, such as NAGs during ctg conversion or priority during abk conversion. You may disable this custom usage using --no-learn as it may confuse other software making use of this field.

[alpltl]

jja-0.3.1 is released today. As of this version jja can write Arena, aka `abk` books. Note, Arena, aka abk, opening book file writing support is only supported from Chessbase, aka ctg books. Use the command line flags --author, --comment, --probability-priority, --probability-games, --probability-win-percent to configure ABK header metadata. Game statistics (minimum number of games/wins, win percentages for both sides) are managed automatically by jja.

In-place editing for Arena opening books is also possible using -i, --in-place=SUFFIX command line option. Conversion from PolyGlot, aka bin, and ChessMaster, aka obk opening books to Arena, aka abk opening book files is planned for a future release.

Install

To compile from source, use

Code: Select all

cargo install jja

. This requires the rust toolchain to be installed.

As an alternative, static builds of jja are hosted on chesswob.org. These versions are signed by GnuPG, using key D076A377FB27DE70. To install, acquire the latest version from chesswob.org, verify the checksum and the GnuPG signature:

Code: Select all

$> curl https://keybase.io/alip/pgp_keys.asc | gpg --import
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13292  100 13292    0     0  13535      0 --:--:-- --:--:-- --:--:-- 26584
gpg: key D076A377FB27DE70: public key "Ali Polatel (Caissa AI) <alip@caissa.ai>" imported
gpg: Total number processed: 1
gpg:               imported: 1
$> for f in jja{,.sha512sum,.sha512sum.asc}; do wget -q https://chesswob.org/jja/${f}; done
$> gpg --verify jja.sha512sum.asc jja.sha512sum
gpg: Signature made Sun Mar 19 20:52:41 2023 CET
gpg:                using RSA key 5DF763560390A149AC6C14C7D076A377FB27DE70
gpg: Good signature from "Ali Polatel (Caissa AI) ...
$> sha512sum -c jja.sha512sum
jja: OK
$> sudo install -m755 jja /usr/local/bin

More from the ChangeLog:

edit learned to calculate & write ABK header game statistics fields
edit learned to convert CTG book files to ABK book files
edit learned --author and --comment to specify metadata for Arena opening books
edit can edit Arena opening book (abk) files in-place with -i, --in-place=SUFFIX
support for writing Arena (abk) opening books
open learned to wrap long ECO opening lines into multiple lines
find no longer panics on some abk books with entries having invalid uci
edit takes move priority into account for weight on abk to bin conversion
match learned --move-selection={best_move,uniform_random,weighted_random} to pick move selection algoritm for book moves
fix castle decoding on polyglot read/query
fix error return when no positions found in abk, obk and ctg find
fix promotion handling in ctg edit
many improvements to ctg find (move coloring & sorting, average statistics)
new merge command to merge two PolyGlot opening books
new match command to arrange book matches with random playouts

[alpltl]

jja-0.3.2 is released today. As of this version jja cross-compiles successfully for Windows and we're distributing 64-bit windows executables for jja too.

Download:
- Linux: jja-0.3.2.bin, sha512sum, sha512sum.asc
- Windows: jja-0.3.2.exe, sha512sum, sha512sum.asc

The Windows version is only briefly tested under Wine. Please test under Windows if you can and let me know if there're any issues.
Finally here is an asciicast of jja in action:


More from the ChangeLog:
[quote]
edit writes the name of the user and jja's version as comment to Arena opening book metadata on ChessBase to Arena opening book conversion, override with --author, --comment
edit displays a unicode chess board in edit tempfile
fix book traversal on Arena opening book to PolyGlot conversion
enable ansi colors when running on windows terminal
drop unixisms, cross-compiles for windows
fix yet another bug with castle decoding on polyglot read/query
edit no longer tries to spawn the default editor if standard output is not a TTY
fix find and edit for Arena opening book reading, move selection is on par with the Arena GUI
new hash command to calculate the Zobrist hash of the given position
fix infinite loop while converting some big Arena opening book files
improve hashing performance by avoiding double hashing using a hasher builder for Zobrist hashes
improve hashing performance using shakmaty crate's Zobrist Hashing implementation rather than the internal one.

[mvanthoor]

Very nice; where did you get the information about how to read/write books such as the one from Arena, ChessMaster and Chessbase?

[alpltl]

Very nice; where did you get the information about how to read/write books such as the one from Arena, ChessMaster and Chessbase?

Thank you, I've added an Acknowledgements section to the readme which gives information about this, see: https://git.sr.ht/~alip/jja#acknowledgements

[alpltl]

We're thrilled to announce the release of jja 0.4.0, an update to our versatile chess file format tool. The latest version introduces the ability to create PolyGlot books from PGN files, including compressed PGNs. It also features improved performance and memory management for handling large PGN files and the ability to filter moves using Filter Expressions for creating specialized opening books.

In addition to these improvements, we've generated five PolyGlot books for the community, which are available for download under the Creative Commons CC0 license. You can use them for research, commercial purposes, publication, or anything else you'd like.

To get started with jja 0.4.0, follow the installation instructions in the original blog post. We look forward to your feedback and are excited to see how these new features will help chess programmers elevate their game. Gens una sumus!

[chesskobra]

In addition to these improvements, we've generated five PolyGlot books for the community, which are available for download under the Creative Commons CC0 license. You can use them for research, commercial purposes, publication, or anything else you'd like.

Where can I find these 5 polyglot books that you mention?
Another question: some time ago I had asked a few questions about getting opening lines in PGN out of an opening book. I was wondering if jja can do this.

[alpltl]

Where can I find these 5 polyglot books that you mention?
Another question: some time ago I had asked a few questions about getting opening lines in PGN out of an opening book. I was wondering if jja can do this.

1. It's mentioned in the blog post. See here: https://www.chesswob.org/jja/books/
2. Not yet, although I plan to do this at some point, I opened an issue in our issue tracker so you can follow progress easily: https://todo.sr.ht/~alip/jja/8

[jefk]

hello Ali

awesome job, i remembered something done earlier,
and found it, so i think you should have a look the program 'openbook'
by Graham o Neil, see his program openbook on his site
http://goneill.co.nz/chess.php
NB unzip and see the readme, source code is included
(it's made in pascal apparently); new thing
with your program is the .ctg (chessbase)feature.
What i found over the years is that there sometimes
are issues with such programs when the book(s) become
too large; but i leave it to you (and Graham o N if he
still would be available); good luck !
PS for people as me a win64 interface would
make it especially awesome but that's
something for later (maybe)