Polyglot 64-bit

[Ozymandias]

So we're clear, can we make a book containing just moves 6-10, with the released executable?

Yep, in the next POLY version.

You don't need a polyglot.ini using --make-book.

Does POLY have the old size limitations or has it also been updated to 64-bit?

Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.

[Rebel]

So we're clear, can we make a book containing just moves 6-10, with the released executable?

Yep, in the next POLY version.

You don't need a polyglot.ini using --make-book.

Does POLY have the old size limitations or has it also been updated to 64-bit?

Remains 32-bit, thus max-size remains 2Gb.

Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.

Cool.

[Michel]

Is it not sufficient to replace all int's in the sources book*.c which obviously refer to some position in the book with uint64?

I may take up maintaining polyglot again. I find it a shame that polyglot book files after all these years still do not contain a header (for copyright messages, variants, etc...).

I created a backwards compatible header format in 2012 and implemented it (including an api)

http://hardy.uhasselt.be/Toga/pgheader- ... eader.html

Here is an example session

Code: Select all

$ ./pgheader -h
pgheader <options> [<file>]
Update a header, adding a default one if necessary
<file>            input file
Options:
-h                print this help 
-l                print the known variant list 
-s                print the header
-S                print the header data
-d                delete the header
-v  <variants>    comma separated list of supported variants
-f                force inclusion of unknown variants
-c  <comment>     free format string, may contain newlines encoded as
                  two character strings "\n"
$ ./pgheader -s performance.bin 
pgheader: No header.
$ ./pgheader -c "Performance.bin with header" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
normal
Comment:
Performance.bin with header
./pgheader -v suicide -c "performance.bin with suicide support" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
suicide
Comment:
performance.bin with suicide support

The header support is contained in the version of polyglot (currently 1.4.71b) I was maintaining (for example merging will take the comments and variants into account). Sadly it is not supported in any GUI that I know of. Which is a shame I think.

The format is actually quite generic and could be used to embed other data in a book (such as learning information) in a backwards compatible way.

[Rebel]

Is it not sufficient to replace all int's in the sources book*.c which obviously refer to some position in the book with uint64?

Possibly. And change fseek to fseeki64.

I may take up maintaining polyglot again.

If you do maybe you can have a look at -make-book when it has FEN strings. It doesn't handle FEN strings with EP squares well.

I find it a shame that polyglot book files after all these years still do not contain a header (for copyright messages, variants, etc...).

I created a backwards compatible header format in 2012 and implemented it (including an api)

http://hardy.uhasselt.be/Toga/pgheader- ... eader.html

Here is an example session

Code: Select all

$ ./pgheader -h
pgheader <options> [<file>]
Update a header, adding a default one if necessary
<file>            input file
Options:
-h                print this help 
-l                print the known variant list 
-s                print the header
-S                print the header data
-d                delete the header
-v  <variants>    comma separated list of supported variants
-f                force inclusion of unknown variants
-c  <comment>     free format string, may contain newlines encoded as
                  two character strings "\n"
$ ./pgheader -s performance.bin 
pgheader: No header.
$ ./pgheader -c "Performance.bin with header" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
normal
Comment:
Performance.bin with header
./pgheader -v suicide -c "performance.bin with suicide support" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
suicide
Comment:
performance.bin with suicide support

The header support is contained in the version of polyglot (currently 1.4.71b) I was maintaining (for example merging will take the comments and variants into account). Sadly it is not supported in any GUI that I know of. Which is a shame I think.

The format is actually quite generic and could be used to embed other data in a book (such as learning information) in a backwards compatible way.

I am new to this.

[Ozymandias]

Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.

Cool.

It looks like it suffers from the same limitation that polyglot_tolerant has: it doesn't use virtual memory. This, added to the normal polyglot behaviour of doubling the reserved amount of memory required, as soon as the previous limit is about to be reached, means that you can't even use 20 out of 32GB (37.5% of all system memory is wasted).

[Rebel]

Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.

Cool.

It looks like it suffers from the same limitation that polyglot_tolerant has: it doesn't use virtual memory. This, added to the normal polyglot behaviour of doubling the reserved amount of memory required, as soon as the previous limit is about to be reached, means that you can't even use 20 out of 32GB (37.5% of all system memory is wasted).

I don't follow, what is polyglot_tolerant?

Secondly, with 32Gb ram (Polyglot likely will take 16Gb of internal work space) it must be able to make a book of 5-6-7Gb in one run, meaning without merge.

[Ozymandias]

what is polyglot_tolerant?

Secondly, with 32Gb ram (Polyglot likely will take 16Gb of internal work space) it must be able to make a book of 5-6-7Gb in one run, meaning without merge.

It's one the other Polyglot 64-bit versions I mentioned in my first post, in this thread.

I will try 64GB, once I get the RMA of one of the sticks back, but that might take a while.

[hgm]

The problem of not wanting to allocate even half the memory was also plaguing my EGT generator. With 4GB of physical memory it would not allow me to allocate more than 1GB! But this is inflicted by the library function for allocation, not by the OS itself. I could get around it by just declaring a static array (uninitialized) of the required size, and instead of calling malloc() just set a pointer to the start of that array.

[lukasmonk]

An alternative, which I am currently studying for my program, so that memory is not a problem is to use a database, such as sqlite, or a mixture of a dictionary/cache in memory and a database, to create an intermediate file, prior to final filtering.
The problem with this solution is that it is much slower, but on the other hand there is no limitation as to the size of the created bin.

[Ozymandias]

An alternative, which I am currently studying for my program, so that memory is not a problem is to use a database, such as sqlite, or a mixture of a dictionary/cache in memory and a database, to create an intermediate file, prior to final filtering.
The problem with this solution is that it is much slower, but on the other hand there is no limitation as to the size of the created bin.

Isn't it easier to allow the program to access windows virtual memory? That way it only uses the disk when out of physical memory.