jja: convert CTG books to PolyGlot format (and more!)

[alpltl]

The jja utility is really useful.
Now I can convert most book formats into something that I want to use.
For years, people have been yearning for this.
Thanks, jja team.

Thank you very much for your kind words. I'm glad to hear you find the program useful.

[alpltl]

Just an observation: maybe it is the size of the CTG book being used that is creating issues?

In fact I have observed something similar when I try to write out pgn from a polyglot bin book. When I tested in on a book called Cerebellum3Merge.bin, which is quite large (nearly 178 MB), the pgn output is surprisingly small. (I had got the book from https://zipproth.de/Brainfish/download/, but that link doesn't load for me anymore.)

This is interesting. Thanks for sharing. I'll need to delve deeper into this to figure out why the pgn ends up smaller than expected. As a rule, however, you should know that the output pgn does not include unreachable (or isolated in polyglot terms) positions/variations with regards to the initial position so it's expected that some entries will be missing from pgn.

I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.

[chesskobra]

I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.

I have done a quick test. Here is some data:

file.bin 200k
polyglot info-book 2884 white lines, 2915 black lines, 299 isolated positions
jja edit -o file.pgn file.bin (before the above fix) 2697 lines (closer to polyglot info-book)
jja edit -o file.pgn file.bin (after the above fix) 68072 lines (why so big?)

I can test more on the weekend, and maybe post my test case on gitlab.

[alpltl]

I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.

I have done a quick test. Here is some data:

file.bin 200k
polyglot info-book 2884 white lines, 2915 black lines, 299 isolated positions
jja edit -o file.pgn file.bin (before the above fix) 2697 lines (closer to polyglot info-book)
jja edit -o file.pgn file.bin (after the above fix) 68072 lines (why so big?)

I can test more on the weekend, and maybe post my test case on gitlab.

This is odd, I'd love to take a look at some samples and dive deeper into what's wrong here. I should admit, jja lacks tests so I rely on all you people here to help me figure out bugs . That said, jja is almost feature-complete, and I can focus more of my time on tests now.

Looks like my intended fix actually broke things. Did you look at the pgn? Are there any duplicates in there? Because in my tests I have ran "pgn-extract -D" on the output pgn and found no duplicates. I think it's best we try converting books with smaller sizes first so it's easier to understand what's going on in the pgn.

Finally always feel free to use the issue tracker to share more of your findings. Thank you very much.

[Jonathan003]

Where can I find the GitLab page?
I like to use GitLab ore GitHub, for suggestions and bug reports.
That's easier to share screenshots and video captures to report bugs, ore demonstrate some wanted functionality.

[Jonathan003]

In the following position the move 4...0-0 is missing.
rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4
In the bin book after converting the little ctg book I shared here.
There are 4 games with the move 4...0-0 in the original CTG book.
I used the latest version of jja.exe you shared here on Talkchess.com

[Jonathan003]

Thanks for testing. It's good to know the main issue with the conversion is fixed. As to the problem you described, I need to delve deeper into this to understand what's going on. My first instinct is the moves are filtered out somehow because they're assigned zero weight. Can you try to run the conversion giving "--null" as an argument to "jja edit" and see if the moves appear in the bin book? It'd also help me greatly if you can provide me a sample ctg book, and list the positions/moves that you know are missing in the bin book. This way I can quickly reproduce the problem and see why this happens and hopefully fix it.

I have tried the "--null" as an argument to "jja edit" but the converted book still miss some moves. In fact I didn't notice any difference yet by comparing the two bin books, (the one with using "--null" as an argument and the one without using "--null" as an argument).
I only noticed a difference in size of the books
2,110 KB when using the argument "--null" and 1,290 KB without using the argument "--null".

[alpltl]

I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.

I have done a quick test. Here is some data:

file.bin 200k
polyglot info-book 2884 white lines, 2915 black lines, 299 isolated positions
jja edit -o file.pgn file.bin (before the above fix) 2697 lines (closer to polyglot info-book)
jja edit -o file.pgn file.bin (after the above fix) 68072 lines (why so big?)

I can test more on the weekend, and maybe post my test case on gitlab.

Here is a small test of mine using varied.bin which is shipped with Scid.

Code: Select all

alip@rosarote:~/org/books/bin|⇒  jja edit varied.bin -o varied.pgn
Creating output PGN file...
Success creating output PGN file.
Generating a tree of all variations in the book...
Success generating a tree of all variations in the book.
Saving the opening tree into the given PGN file...
Saving done.
alip@rosarote:~/org/books/bin|⇒  pgn-extract -s -D varied.pgn > varied-nodup.pgn
alip@rosarote:~/org/books/bin|⇒  grep -F "[Event " -c varied.pgn varied-nodup.pgn
varied.pgn:384140
varied-nodup.pgn:384140
alip@rosarote:~/org/books/bin|⇒

There are no duplicate lines in the pgn generated by jja.

[alpltl]

Thanks for testing. It's good to know the main issue with the conversion is fixed. As to the problem you described, I need to delve deeper into this to understand what's going on. My first instinct is the moves are filtered out somehow because they're assigned zero weight. Can you try to run the conversion giving "--null" as an argument to "jja edit" and see if the moves appear in the bin book? It'd also help me greatly if you can provide me a sample ctg book, and list the positions/moves that you know are missing in the bin book. This way I can quickly reproduce the problem and see why this happens and hopefully fix it.

I have tried the "--null" as an argument to "jja edit" but the converted book still miss some moves. In fact I didn't notice any difference yet by comparing the two bin books, (the one with using "--null" as an argument and the one without using "--null" as an argument).
I only noticed a difference in size of the books
2,110 KB when using the argument "--null" and 1,290 KB without using the argument "--null".

Well, that's probably a feature not a bug. In fact what's important in PolyGlot books are the books with high weight. And if jja does not have information about how to set a weight from a CTG entry then there's little we can do rather than assigning 0 weight. It's up to the user to preserve these moves using --null. That said I'd love to debug this issue further and see why entries are assigned zero weight. I'll be happy to be proven wrong that there's a bug somewhere. Note also that we're working on a reverse-engineered specification and there're many unclear points, so it's a bit like shooting in the dark .

[chesskobra]

Could you take a look at this test case I created: https://gitlab.com/beejaganita/jja-tests/