Thank you very much for your kind words. I'm glad to hear you find the program useful.Dann Corbit wrote: ↑Sun Jun 11, 2023 2:34 am The jja utility is really useful.
Now I can convert most book formats into something that I want to use.
For years, people have been yearning for this.
Thanks, jja team.
jja: convert CTG books to PolyGlot format (and more!)
Moderators: hgm, Rebel, chrisw
-
- Posts: 57
- Joined: Tue Mar 14, 2023 3:04 pm
- Location: Berlin
- Full name: Ali Polatel
Re: jja: convert CTG books to PolyGlot format (and more!)
Caissa-AI, Caissa-Test, and Caissa-X on LiChess
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
-
- Posts: 57
- Joined: Tue Mar 14, 2023 3:04 pm
- Location: Berlin
- Full name: Ali Polatel
Re: jja: convert CTG books to PolyGlot format (and more!)
I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.alpltl wrote: ↑Fri Jun 09, 2023 11:54 pmThis is interesting. Thanks for sharing. I'll need to delve deeper into this to figure out why the pgn ends up smaller than expected. As a rule, however, you should know that the output pgn does not include unreachable (or isolated in polyglot terms) positions/variations with regards to the initial position so it's expected that some entries will be missing from pgn.chesskobra wrote: ↑Thu Jun 08, 2023 6:27 pmIn fact I have observed something similar when I try to write out pgn from a polyglot bin book. When I tested in on a book called Cerebellum3Merge.bin, which is quite large (nearly 178 MB), the pgn output is surprisingly small. (I had got the book from https://zipproth.de/Brainfish/download/, but that link doesn't load for me anymore.)
Caissa-AI, Caissa-Test, and Caissa-X on LiChess
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
-
- Posts: 260
- Joined: Thu Jul 21, 2022 12:30 am
- Full name: Chesskobra
Re: jja: convert CTG books to PolyGlot format (and more!)
I have done a quick test. Here is some data:alpltl wrote: ↑Tue Jun 13, 2023 9:35 pm
I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.
file.bin 200k
polyglot info-book 2884 white lines, 2915 black lines, 299 isolated positions
jja edit -o file.pgn file.bin (before the above fix) 2697 lines (closer to polyglot info-book)
jja edit -o file.pgn file.bin (after the above fix) 68072 lines (why so big?)
I can test more on the weekend, and maybe post my test case on gitlab.
-
- Posts: 57
- Joined: Tue Mar 14, 2023 3:04 pm
- Location: Berlin
- Full name: Ali Polatel
Re: jja: convert CTG books to PolyGlot format (and more!)
This is odd, I'd love to take a look at some samples and dive deeper into what's wrong here. I should admit, jja lacks tests so I rely on all you people here to help me figure out bugs . That said, jja is almost feature-complete, and I can focus more of my time on tests now.chesskobra wrote: ↑Wed Jun 14, 2023 2:23 amI have done a quick test. Here is some data:alpltl wrote: ↑Tue Jun 13, 2023 9:35 pm
I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.
file.bin 200k
polyglot info-book 2884 white lines, 2915 black lines, 299 isolated positions
jja edit -o file.pgn file.bin (before the above fix) 2697 lines (closer to polyglot info-book)
jja edit -o file.pgn file.bin (after the above fix) 68072 lines (why so big?)
I can test more on the weekend, and maybe post my test case on gitlab.
Looks like my intended fix actually broke things. Did you look at the pgn? Are there any duplicates in there? Because in my tests I have ran "pgn-extract -D" on the output pgn and found no duplicates. I think it's best we try converting books with smaller sizes first so it's easier to understand what's going on in the pgn.
Finally always feel free to use the issue tracker to share more of your findings. Thank you very much.
Caissa-AI, Caissa-Test, and Caissa-X on LiChess
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
-
- Posts: 239
- Joined: Fri Jul 06, 2018 4:23 pm
- Full name: Jonathan Cremers
Re: jja: convert CTG books to PolyGlot format (and more!)
Where can I find the GitLab page?
I like to use GitLab ore GitHub, for suggestions and bug reports.
That's easier to share screenshots and video captures to report bugs, ore demonstrate some wanted functionality.
I like to use GitLab ore GitHub, for suggestions and bug reports.
That's easier to share screenshots and video captures to report bugs, ore demonstrate some wanted functionality.
-
- Posts: 239
- Joined: Fri Jul 06, 2018 4:23 pm
- Full name: Jonathan Cremers
Re: jja: convert CTG books to PolyGlot format (and more!)
In the following position the move 4...0-0 is missing.
rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4
In the bin book after converting the little ctg book I shared here.
There are 4 games with the move 4...0-0 in the original CTG book.
I used the latest version of jja.exe you shared here on Talkchess.com
rnbqk2r/pppp1ppp/4pn2/8/1bPP4/5N2/PP1NPPPP/R1BQKB1R b KQkq - 0 4
In the bin book after converting the little ctg book I shared here.
There are 4 games with the move 4...0-0 in the original CTG book.
I used the latest version of jja.exe you shared here on Talkchess.com
-
- Posts: 239
- Joined: Fri Jul 06, 2018 4:23 pm
- Full name: Jonathan Cremers
Re: jja: convert CTG books to PolyGlot format (and more!)
I have tried the "--null" as an argument to "jja edit" but the converted book still miss some moves. In fact I didn't notice any difference yet by comparing the two bin books, (the one with using "--null" as an argument and the one without using "--null" as an argument).Thanks for testing. It's good to know the main issue with the conversion is fixed. As to the problem you described, I need to delve deeper into this to understand what's going on. My first instinct is the moves are filtered out somehow because they're assigned zero weight. Can you try to run the conversion giving "--null" as an argument to "jja edit" and see if the moves appear in the bin book? It'd also help me greatly if you can provide me a sample ctg book, and list the positions/moves that you know are missing in the bin book. This way I can quickly reproduce the problem and see why this happens and hopefully fix it.
I only noticed a difference in size of the books
2,110 KB when using the argument "--null" and 1,290 KB without using the argument "--null".
-
- Posts: 57
- Joined: Tue Mar 14, 2023 3:04 pm
- Location: Berlin
- Full name: Ali Polatel
Re: jja: convert CTG books to PolyGlot format (and more!)
Here is a small test of mine using varied.bin which is shipped with Scid.chesskobra wrote: ↑Wed Jun 14, 2023 2:23 amI have done a quick test. Here is some data:alpltl wrote: ↑Tue Jun 13, 2023 9:35 pm
I have done some work on this issue, and fixed a bug which caused valid lines to be skipped in pgn output. See the commit if you're interested in the details. There is still more to be done to ensure full compatibility with polyglot wrt. isolated positions. I'll post more updates when I fix more things. Meanwhile, I'd be very happy if you could test the latest git build (linux-glibc, linux-musl, and windows), with various polyglot books and see if you can figure out which lines exactly are missing in the pgn output. Thanks in advance.
file.bin 200k
polyglot info-book 2884 white lines, 2915 black lines, 299 isolated positions
jja edit -o file.pgn file.bin (before the above fix) 2697 lines (closer to polyglot info-book)
jja edit -o file.pgn file.bin (after the above fix) 68072 lines (why so big?)
I can test more on the weekend, and maybe post my test case on gitlab.
Code: Select all
alip@rosarote:~/org/books/bin|⇒ jja edit varied.bin -o varied.pgn
Creating output PGN file...
Success creating output PGN file.
Generating a tree of all variations in the book...
Success generating a tree of all variations in the book.
Saving the opening tree into the given PGN file...
Saving done.
alip@rosarote:~/org/books/bin|⇒ pgn-extract -s -D varied.pgn > varied-nodup.pgn
alip@rosarote:~/org/books/bin|⇒ grep -F "[Event " -c varied.pgn varied-nodup.pgn
varied.pgn:384140
varied-nodup.pgn:384140
alip@rosarote:~/org/books/bin|⇒
Caissa-AI, Caissa-Test, and Caissa-X on LiChess
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
-
- Posts: 57
- Joined: Tue Mar 14, 2023 3:04 pm
- Location: Berlin
- Full name: Ali Polatel
Re: jja: convert CTG books to PolyGlot format (and more!)
Well, that's probably a feature not a bug. In fact what's important in PolyGlot books are the books with high weight. And if jja does not have information about how to set a weight from a CTG entry then there's little we can do rather than assigning 0 weight. It's up to the user to preserve these moves using --null. That said I'd love to debug this issue further and see why entries are assigned zero weight. I'll be happy to be proven wrong that there's a bug somewhere. Note also that we're working on a reverse-engineered specification and there're many unclear points, so it's a bit like shooting in the dark .Jonathan003 wrote: ↑Wed Jun 14, 2023 11:24 amI have tried the "--null" as an argument to "jja edit" but the converted book still miss some moves. In fact I didn't notice any difference yet by comparing the two bin books, (the one with using "--null" as an argument and the one without using "--null" as an argument).Thanks for testing. It's good to know the main issue with the conversion is fixed. As to the problem you described, I need to delve deeper into this to understand what's going on. My first instinct is the moves are filtered out somehow because they're assigned zero weight. Can you try to run the conversion giving "--null" as an argument to "jja edit" and see if the moves appear in the bin book? It'd also help me greatly if you can provide me a sample ctg book, and list the positions/moves that you know are missing in the bin book. This way I can quickly reproduce the problem and see why this happens and hopefully fix it.
I only noticed a difference in size of the books
2,110 KB when using the argument "--null" and 1,290 KB without using the argument "--null".
Caissa-AI, Caissa-Test, and Caissa-X on LiChess
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
ChessWoB: Chess without Boundaries
jja: Jin, Jîyan, Azadî!
Follow @alip on Mastodon!
-
- Posts: 260
- Joined: Thu Jul 21, 2022 12:30 am
- Full name: Chesskobra
Re: jja: convert CTG books to PolyGlot format (and more!)
Could you take a look at this test case I created: https://gitlab.com/beejaganita/jja-tests/