jja: convert CTG books to PolyGlot format (and more!)

[alpltl]

jja-0.9.1 has been released.

## Download

* Windows: jja-0.9.1.exe, sha512sum, signature
* Linux GLibc: jja-0.9.1-glibc.bin, sha512sum, signature
* Linux Musl: jja-0.9.1-musl.bin, sha512sum, signature

To install from source, use "cargo install jja".

## ChangeLog

- upgrade nix crate from 0.27 to 0.28.
- upgrade tempfile crate from 3.8 to 3.10.
- upgrade rust-embed crate from 8.1 to 8.4.
- upgrade rusqlite crate from 0.30 to 0.31.
- upgrade rayon crate from 1.8 to 1.10.
- upgrade hostname crate from 0.3 to 0.4.
- Add support for restoring BrainLearn experience files from Lichess
evaluations export JSON.
- TotalPlyCount PGN tag is now supported in make filters.
- fix --win,draw,loss-factor calculation for make
- Add 50-moves detection and 3-position repetition detection to the play
subcommand.
- upgrade once_cell crate from 1.18 to 1.19.
- upgrade rayon crate from 1.7 to 1.8.
- upgrade zstd crate from 0.12 to 0.13.
- upgrade rusqlite crate from 0.29 to 0.30.
- upgrade dialoguer crate from 0.10 to 0.11.
- upgrade i18n-embed crate from 0.13 to 0.14.
- upgrade rust-embed crate from 6.8 to 8.0.
- upgrade built crate from 0.6 to 0.7.
- fix quote matching at word boundaries when no wildcard characters are given.
- merge learned -O<u16>, --outlier-threshold=<u16> to specify an outlier threshold
where moves with weights higher than the given threshold are filtered out from the
merged output book. This stands from the fact that if the weight differences of
the same move in two different books have a high difference, calculating any
average may not yield meaningful results. The default is 0, which disables this
feature.
- merge learned --strategy dynmid which stands for
jja::merge::MergeStrategy::DynamicMidpoint.
- merge learned --strategy wdist which stands for
jja::merge::MergeStrategy::WeightedDistance.
- upgrade nix crate from 0.26 to 0.27 and add the signal feature.
- merge learned --strategy wmedian which stands for
jja::merge::MergeStrategy::WeightedMedian.
- merge learned --strategy entropy which stands for
jja::merge::MergeStrategy::Entropy.
- merge learned --strategy quadratic which stands for
jja::merge::MergeStrategy::QuadraticMean. This strategy merges using the Quadratic
Mean (Root Mean Square) approach. The Quadratic Mean (also known as the Root Mean
Square) is a statistical measure of the magnitude of a set of numbers. It offers a
more balanced view, especially when dealing with numbers of varying magnitudes.
- merge learned --strategy lavg which stands for
jja::merge::MergeStrategy::LogarithmicAverage. This strategy merges using a
logarithmic averaging approach. Given that logarithmic functions compress large
values and expand small values, we can use them to get a merge strategy that's
sensitive to differences in smaller weights while being more resistant to
disparities in larger weights.

[alpltl]

I tried the ctg->abk conversion with the following command:

Code: Select all

jja edit Book.ctg -o Book.abk -C -N --author=amchess --comment=amchessBook --probability-priority=2 --probability-games=15 --probability-win-percent=0

The book is created, but the probabilities in Arena are totally wrong (decreasing order) and overall the book is unusable with an engine (demo mode, for example), contrary to ohter abk books.
In what I'm wrong?

Interesting. First of all thank you very much for testing and your valuable feedback. To help me diagnose the issue better can you send me the output of "jja find" for a few positions from the CTG book and corresponding entries in the ABK book where you spotted that the probabilities are wrong? You can use "jja find" with a FEN or a PGN line, e.g: "jja find -p'e4 c6' book.ctg" or jja find -f<FEN> book.abk".

[Ajaja]

Does anybody know how to convert a CTG file to a PolyGlot BIN file with weights that are equivalent to the [%] values calculated in Fritz with Optimal book settings?
https://help.chessbase.com/Fritz/19/Eng ... 000034.htm

The value on the right [%] is the practical probability of the program playing the move. This is a product of the statistical evaluation and the weights. If you change the latter you can see how the [%] value (but not the “Prob”) changes. In Book options you can determine how much the weights influence the probability of play.

[Ajaja]

For example:

Code: Select all

D:\Temp>D:\Games\Chess\Jja\jja-0.9.1.exe find -p "Nf3 d5 c4 d4 b4" "f.ctg"
+---+------+-----+-----+------+------+---------+-------------+----------------+
| * | UCI  | NAG | Win | Draw | Loss | Average | Performance | Recommendation |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 1 | f7f6 |     | 29  | 28   | 32   | 2619.31 | 2655.94     | 0x80           |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 2 | c8g4 |     | 28  | 14   | 25   | 2611.54 | 2644.88     | 0x80           |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 3 | c7c5 | ?   | 41  | 30   | 48   | 2619.04 | 2664.49     | 0x80           |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 4 | g7g5 | ?   | 18  | 7    | 26   | 2602.04 | 2602.18     | 0x80           |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 5 | g7g6 | ?   | 20  | 10   | 5    | 2594.12 | 2573.56     | 0x0            |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 6 | a7a5 | ?   | 8   | 1    | 0    | 2609.57 | 2551.00     | 0x0            |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 7 | g8f6 | ?   | 2   | 2    | 4    | 2587.25 | 2653.12     | 0x0            |
+---+------+-----+-----+------+------+---------+-------------+----------------+
| 8 | e7e6 | ?   | 0   | 0    | 1    | 2505.00 | 2600.00     | 0x0            |
+---+------+-----+-----+------+------+---------+-------------+----------------+

Code: Select all

D:\Temp>D:\Games\Chess\Jja\jja-0.9.1.exe edit --output "f.bin" --no-learn "f.ctg"
Creating output PolyGlot opening book...
Success creating output PolyGlot opening book.
Searching for move entries in the CTG input file. This may take a while.
Skipping illegal move with fen:r3kbnr/pQp2ppp/8/3p4/8/2Pn1P2/PPKP2PP/RNB1qB1R b kq - move:256
Skipping illegal move with fen:rn1qkb1Q/p3pp1p/2p5/5b2/2pP4/5N2/P3BPPP/q1BQK2R w Kq - move:4079
Skipping illegal move with fen:rnb1Q3/pp3qpp/6k1/2pP1pPn/4p2P/2N5/PP3P2/R3K2R w KQ - move:3896
Skipping illegal move with fen:r3k2r/pp3p2/2n5/4P2p/2Pp1PpN/6K1/PP3QPP/RNB1q3 b kq - move:256
Skipping illegal move with fen:Q1bqk2r/p3bppp/5n2/2Pp4/5B2/2P5/P3PP1P/RN1QKB1q b Qk - move:471
Skipping illegal move with fen:rnb1Qb1r/ppkp2pp/2pN1p2/8/3P4/8/PqP2PPP/R3KBNR w KQ - move:3896
Found 9114991 positions in the CTG input file.
Saving 9114991 positions from input book to the output book...
Success copying entries from the input book to the output book.
Saving done.    [9114991/9114991] 239,357.5388/s [=====================================================================================================================] [00:00:38] ETA 00:00:00
# jja Chess File Information
Path: f.bin
Size: 38.78 MB (38780928 bytes)
Type: BIN
Sha256: da72ffcc278f01041fca48fd48e5057be733c70e10be70db519f0f73b877e355
## jja PolyGlot File Information
Number of entries: 2423808
Size of a single entry: 16 bytes

Code: Select all

D:\Temp>D:\Games\Chess\Jja\jja-0.9.1.exe find -p "Nf3 d5 c4 d4 b4" "f.bin"
+---+------+--------+-------+
| * | UCI  | Weight | Learn |
+---+------+--------+-------+
| 1 | c7c5 | 65520  | 0     |
+---+------+--------+-------+
| 2 | g7g5 | 65451  | 0     |
+---+------+--------+-------+
| 3 | f7f6 | 65233  | 0     |
+---+------+--------+-------+
| 4 | c8g4 | 65218  | 0     |
+---+------+--------+-------+
| 5 | g7g6 | 545    | 0     |
+---+------+--------+-------+
| 6 | a7a5 | 513    | 0     |
+---+------+--------+-------+
| 7 | g8f6 | 501    | 0     |
+---+------+--------+-------+
| 8 | e7e6 | 495    | 0     |
+---+------+--------+-------+

Those weights in f.bin do not correspond to the calculated [%] values from Fritz 20 with Optimal book settings at all: