Polyglot 64-bit

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Ozymandias
Posts: 1096
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Fri Jun 14, 2019 7:35 pm

Rebel wrote:
Fri Jun 14, 2019 7:19 pm
Ozymandias wrote:
Thu Jun 13, 2019 7:50 pm
So we're clear, can we make a book containing just moves 6-10, with the released executable?
Yep, in the next POLY version.

You don't need a polyglot.ini using --make-book.
Does POLY have the old size limitations or has it also been updated to 64-bit?

Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.

User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot 64-bit

Post by Rebel » Sat Jun 15, 2019 6:50 am

Ozymandias wrote:
Fri Jun 14, 2019 7:35 pm
Rebel wrote:
Fri Jun 14, 2019 7:19 pm
Ozymandias wrote:
Thu Jun 13, 2019 7:50 pm
So we're clear, can we make a book containing just moves 6-10, with the released executable?
Yep, in the next POLY version.

You don't need a polyglot.ini using --make-book.
Does POLY have the old size limitations or has it also been updated to 64-bit?
Remains 32-bit, thus max-size remains 2Gb.
Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.
Cool.
Everybody is unique, except me.

Michel
Posts: 2038
Joined: Sun Sep 28, 2008 11:50 pm

Re: Polyglot 64-bit

Post by Michel » Sat Jun 15, 2019 8:19 am

Is it not sufficient to replace all int's in the sources book*.c which obviously refer to some position in the book with uint64?

I may take up maintaining polyglot again. I find it a shame that polyglot book files after all these years still do not contain a header (for copyright messages, variants, etc...).

I created a backwards compatible header format in 2012 and implemented it (including an api)

http://hardy.uhasselt.be/Toga/pgheader- ... eader.html

Here is an example session

Code: Select all

$ ./pgheader -h
pgheader <options> [<file>]
Update a header, adding a default one if necessary
<file>            input file
Options:
-h                print this help 
-l                print the known variant list 
-s                print the header
-S                print the header data
-d                delete the header
-v  <variants>    comma separated list of supported variants
-f                force inclusion of unknown variants
-c  <comment>     free format string, may contain newlines encoded as
                  two character strings "\n"
$ ./pgheader -s performance.bin 
pgheader: No header.
$ ./pgheader -c "Performance.bin with header" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
normal
Comment:
Performance.bin with header
./pgheader -v suicide -c "performance.bin with suicide support" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
suicide
Comment:
performance.bin with suicide support
The header support is contained in the version of polyglot (currently 1.4.71b) I was maintaining (for example merging will take the comments and variants into account). Sadly it is not supported in any GUI that I know of. Which is a shame I think.

The format is actually quite generic and could be used to embed other data in a book (such as learning information) in a backwards compatible way.
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot 64-bit

Post by Rebel » Sat Jun 15, 2019 6:31 pm

Michel wrote:
Sat Jun 15, 2019 8:19 am
Is it not sufficient to replace all int's in the sources book*.c which obviously refer to some position in the book with uint64?
Possibly. And change fseek to fseeki64.
Michel wrote:
Sat Jun 15, 2019 8:19 am
I may take up maintaining polyglot again.
If you do maybe you can have a look at -make-book when it has FEN strings. It doesn't handle FEN strings with EP squares well.
Michel wrote:
Sat Jun 15, 2019 8:19 am
I find it a shame that polyglot book files after all these years still do not contain a header (for copyright messages, variants, etc...).

I created a backwards compatible header format in 2012 and implemented it (including an api)

http://hardy.uhasselt.be/Toga/pgheader- ... eader.html

Here is an example session

Code: Select all

$ ./pgheader -h
pgheader <options> [<file>]
Update a header, adding a default one if necessary
<file>            input file
Options:
-h                print this help 
-l                print the known variant list 
-s                print the header
-S                print the header data
-d                delete the header
-v  <variants>    comma separated list of supported variants
-f                force inclusion of unknown variants
-c  <comment>     free format string, may contain newlines encoded as
                  two character strings "\n"
$ ./pgheader -s performance.bin 
pgheader: No header.
$ ./pgheader -c "Performance.bin with header" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
normal
Comment:
Performance.bin with header
./pgheader -v suicide -c "performance.bin with suicide support" performance.bin
$ ./pgheader -s performance.bin
Variants supported:
suicide
Comment:
performance.bin with suicide support
The header support is contained in the version of polyglot (currently 1.4.71b) I was maintaining (for example merging will take the comments and variants into account). Sadly it is not supported in any GUI that I know of. Which is a shame I think.

The format is actually quite generic and could be used to embed other data in a book (such as learning information) in a backwards compatible way.
I am new to this.
Everybody is unique, except me.

User avatar
Ozymandias
Posts: 1096
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Thu Jun 20, 2019 5:54 pm

Rebel wrote:
Sat Jun 15, 2019 6:50 am
Ozymandias wrote:
Fri Jun 14, 2019 7:35 pm
Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.
Cool.
It looks like it suffers from the same limitation that polyglot_tolerant has: it doesn't use virtual memory. This, added to the normal polyglot behaviour of doubling the reserved amount of memory required, as soon as the previous limit is about to be reached, means that you can't even use 20 out of 32GB (37.5% of all system memory is wasted).

User avatar
Rebel
Posts: 4541
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot 64-bit

Post by Rebel » Thu Jun 20, 2019 9:31 pm

Ozymandias wrote:
Thu Jun 20, 2019 5:54 pm
Rebel wrote:
Sat Jun 15, 2019 6:50 am
Ozymandias wrote:
Fri Jun 14, 2019 7:35 pm
Yes, I've been finally been able to run it. Next week I'm getting 32GB more of RAM, I'll test the upper limits of this thing then.
Cool.
It looks like it suffers from the same limitation that polyglot_tolerant has: it doesn't use virtual memory. This, added to the normal polyglot behaviour of doubling the reserved amount of memory required, as soon as the previous limit is about to be reached, means that you can't even use 20 out of 32GB (37.5% of all system memory is wasted).
I don't follow, what is polyglot_tolerant?

Secondly, with 32Gb ram (Polyglot likely will take 16Gb of internal work space) it must be able to make a book of 5-6-7Gb in one run, meaning without merge.
Everybody is unique, except me.

User avatar
Ozymandias
Posts: 1096
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Fri Jun 21, 2019 11:18 am

Rebel wrote:
Thu Jun 20, 2019 9:31 pm
what is polyglot_tolerant?

Secondly, with 32Gb ram (Polyglot likely will take 16Gb of internal work space) it must be able to make a book of 5-6-7Gb in one run, meaning without merge.
It's one the other Polyglot 64-bit versions I mentioned in my first post, in this thread.

I will try 64GB, once I get the RMA of one of the sticks back, but that might take a while.

User avatar
hgm
Posts: 23483
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: Polyglot 64-bit

Post by hgm » Fri Jun 21, 2019 11:48 am

The problem of not wanting to allocate even half the memory was also plaguing my EGT generator. With 4GB of physical memory it would not allow me to allocate more than 1GB! But this is inflicted by the library function for allocation, not by the OS itself. I could get around it by just declaring a static array (uninitialized) of the required size, and instead of calling malloc() just set a pointer to the start of that array.

lukasmonk
Posts: 44
Joined: Sun Feb 07, 2010 6:35 pm
Location: Spain
Contact:

Re: Polyglot 64-bit

Post by lukasmonk » Sun Jun 23, 2019 10:10 am

An alternative, which I am currently studying for my program, so that memory is not a problem is to use a database, such as sqlite, or a mixture of a dictionary/cache in memory and a database, to create an intermediate file, prior to final filtering.
The problem with this solution is that it is much slower, but on the other hand there is no limitation as to the size of the created bin.

User avatar
Ozymandias
Posts: 1096
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Sun Jun 23, 2019 12:30 pm

lukasmonk wrote:
Sun Jun 23, 2019 10:10 am
An alternative, which I am currently studying for my program, so that memory is not a problem is to use a database, such as sqlite, or a mixture of a dictionary/cache in memory and a database, to create an intermediate file, prior to final filtering.
The problem with this solution is that it is much slower, but on the other hand there is no limitation as to the size of the created bin.
Isn't it easier to allow the program to access windows virtual memory? That way it only uses the disk when out of physical memory.

Post Reply