pgn to polyglot

[flok]

Hello,

I've got a 350 GB pgn file that I'd like to convert to a polyglot-book.
My laptop has 64 GB ram but for the polyglot-program (the original) that is not enough to do so.
Does anyone know a program (open source so that I can run it on Linux) that can handle such large files? Preferably read from a pipe so that I don't have to decompress it first (polyglot can do so).


regards

[jefk]

have a look at "Polyglot Tolerant"

https://chess.massimilianogoi.com/downl ... ttolerant/

https://askubuntu.com/questions/1316314 ... file-forma

[flok]

have a look at "Polyglot Tolerant"

https://chess.massimilianogoi.com/downl ... ttolerant/

https://askubuntu.com/questions/1316314 ... file-forma

I looked at polyglot-tolerant and it looks like to be polyglot with a few tweaks but not for memory management.

[Jim Ablett]

You could try Windows program PGN Book Creator. (may work with Wine)

I haven't tested with very large pgns.

Download my version with doesn't trigger antivirus. Original uses Python script converted to executable which will normally trigger antiviruses but mine is compressed with warp packer which doesn't trigger alerts.

https://drive.proton.me/urls/6WSYHGE5A8#406s0ye6GJ0z

Jim.

[Dann Corbit]

JJA converts every imaginable book format
https://crates.io/crates/jja

[pohl4711]

I have an old tool on my googledrive, which allows to convert Arena books into polyglot books (abk2bin is the name)
So, if you manage to build your book in Arena, it should be easy convertable into polyglot, using this tool.

[Rebel]

Assuming you use polyglot from Fabien L, split the PGN into x parts your memory can handle, then merge them together.

But you need the 64 bit version of Polyglot because of the 2Gb filesize limit of a 32 bit application.

[flok]

Assuming you use polyglot from Fabien L, split the PGN into x parts your memory can handle, then merge them together.

But you need the 64 bit version of Polyglot because of the 2Gb filesize limit of a 32 bit application.

But won't that affect the weights-variable incorrectly? I'm going to dig through polyglot I to see how it handles the merges.

[...]

Code: Select all

               for(j=0; j<n; j++) {
                  if(e[i].move == e[j].move) {
                     e[j].count += e[i--].count;
                     break;
                  }
               }

That is what I was afraid off: counts are just added with no overflow check or anything.
With large sets this will fail.
So I wrote something that handles this and can merge more than 2 files in one go. I'll publish it tomorrow.

[Rebel]

Assuming you use polyglot from Fabien L, split the PGN into x parts your memory can handle, then merge them together.

But you need the 64 bit version of Polyglot because of the 2Gb filesize limit of a 32 bit application.

But won't that affect the weights-variable incorrectly? I'm going to dig through polyglot I to see how it handles the merges.

[...]

Code: Select all

               for(j=0; j<n; j++) {
                  if(e[i].move == e[j].move) {
                     e[j].count += e[i--].count;
                     break;
                  }
               }

That is what I was afraid off: counts are just added with no overflow check or anything.
With large sets this will fail.
So I wrote something that handles this and can merge more than 2 files in one go. I'll publish it tomorrow.

Polyglot itself has a --merge function.

[flok]

Assuming you use polyglot from Fabien L, split the PGN into x parts your memory can handle, then merge them together.

But you need the 64 bit version of Polyglot because of the 2Gb filesize limit of a 32 bit application.

But won't that affect the weights-variable incorrectly? I'm going to dig through polyglot I to see how it handles the merges.

[...]

Code: Select all

               for(j=0; j<n; j++) {
                  if(e[i].move == e[j].move) {
                     e[j].count += e[i--].count;
                     break;
                  }
               }

That is what I was afraid off: counts are just added with no overflow check or anything.
With large sets this will fail.
So I wrote something that handles this and can merge more than 2 files in one go. I'll publish it tomorrow.

Polyglot itself has a --merge function.

That broken code *is* from polyglot.