Polyglot 64-bit

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4463
Joined: Thu Aug 18, 2011 10:04 am

Polyglot 64-bit

Post by Rebel » Thu Jun 13, 2019 5:56 am

Polyglot (as far as I can tell) always has been 32-bit which limited the working space during --make-book to 1Gb. Compiling Polyglot as a 64-bit program takes this obstacle away. I could make a Polyglot book of 1.5Gb in one run while Polyglot workspace increased to 5.5Gb of memory.

Not only is this more comfortable it also has the advantage you don't need --merge-book any longer which enables you to produce better books because --merge-book has the (nasty) habit not to recalculate the weights.

Caution - Now that you can make books as big as your memory allows you take note that when a created Polyglot book exceeds the size of 2Gb it can't be used by engines that access a Polyglot book with traditional 32-bit instructions, some of the Polyglot code needs an update first.

Download at the end of the page - http://rebel13.nl/download/books.html
Everybody is unique, except me.

Ferdy
Posts: 3933
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Polyglot 64-bit

Post by Ferdy » Thu Jun 13, 2019 8:05 am

Thanks Ed, successfully created a book from not less than 600k games.

Book 2 in the gui is from polyglot book built from polyglot64.

Image

User avatar
Ozymandias
Posts: 1071
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Thu Jun 13, 2019 8:22 am

Rebel wrote:
Thu Jun 13, 2019 5:56 am
Polyglot (as far as I can tell) always has been 32-bit which limited the working space during --make-book to 1Gb. Compiling Polyglot as a 64-bit program takes this obstacle away. I could make a Polyglot book of 1.5Gb in one run while Polyglot workspace increased to 5.5Gb of memory.

Not only is this more comfortable it also has the advantage you don't need --merge-book any longer which enables you to produce better books because --merge-book has the (nasty) habit not to recalculate the weights.

Caution - Now that you can make books as big as your memory allows you take note that when a created Polyglot book exceeds the size of 2Gb it can't be used by engines that access a Polyglot book with traditional 32-bit instructions, some of the Polyglot code needs an update first.

Download at the end of the page - http://rebel13.nl/download/books.html
Looking forward to breaking the 2Gb limit. 64 versions already existed, the first one was posted in this forum years ago, and there's also the so-called "tolerant" version. Both failed to address that size limitation.

As for the --merge-book option, I always liked it as is. I will try your Polyglot 64-bit as soon as I upgrade the memory subsystem (prices getting finally good).

User avatar
Rebel
Posts: 4463
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot 64-bit

Post by Rebel » Thu Jun 13, 2019 9:11 am

Ozymandias wrote:
Thu Jun 13, 2019 8:22 am
Rebel wrote:
Thu Jun 13, 2019 5:56 am
Polyglot (as far as I can tell) always has been 32-bit which limited the working space during --make-book to 1Gb. Compiling Polyglot as a 64-bit program takes this obstacle away. I could make a Polyglot book of 1.5Gb in one run while Polyglot workspace increased to 5.5Gb of memory.

Not only is this more comfortable it also has the advantage you don't need --merge-book any longer which enables you to produce better books because --merge-book has the (nasty) habit not to recalculate the weights.

Caution - Now that you can make books as big as your memory allows you take note that when a created Polyglot book exceeds the size of 2Gb it can't be used by engines that access a Polyglot book with traditional 32-bit instructions, some of the Polyglot code needs an update first.

Download at the end of the page - http://rebel13.nl/download/books.html
Looking forward to breaking the 2Gb limit. 64 versions already existed, the first one was posted in this forum years ago, and there's also the so-called "tolerant" version. Both failed to address that size limitation.

As for the --merge-book option, I always liked it as is. I will try your Polyglot 64-bit as soon as I upgrade the memory subsystem (prices getting finally good).
To bypass the 2Gb limit I made an option in ProDeo to access 4 books but only 1 search is needed.
Book-1 for move 0-5
Book-2 for move 6-10
Book-3 for move 11-15
Book-4 moves >=16

So in theory I can make a book close to 8Gb.
Everybody is unique, except me.

User avatar
Ozymandias
Posts: 1071
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Thu Jun 13, 2019 9:22 am

Rebel wrote:
Thu Jun 13, 2019 9:11 am
To bypass the 2Gb limit I made an option in ProDeo to access 4 books but only 1 search is needed.
Book-1 for move 0-5
Book-2 for move 6-10
Book-3 for move 11-15
Book-4 moves >=16

So in theory I can make a book close to 8Gb.
Even if your engine of choice offered the chance to address so many books, which they don't, wouldn't books 2-4 need to include the first moves? There's a "-max-ply" opinion, but to do what you suggest, you'd also need a "-min-ply" one.

User avatar
Rebel
Posts: 4463
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot 64-bit

Post by Rebel » Thu Jun 13, 2019 3:38 pm

Ozymandias wrote:
Thu Jun 13, 2019 9:22 am
Rebel wrote:
Thu Jun 13, 2019 9:11 am
To bypass the 2Gb limit I made an option in ProDeo to access 4 books but only 1 search is needed.
Book-1 for move 0-5
Book-2 for move 6-10
Book-3 for move 11-15
Book-4 moves >=16

So in theory I can make a book close to 8Gb.
Even if your engine of choice offered the chance to address so many books, which they don't, wouldn't books 2-4 need to include the first moves? There's a "-max-ply" opinion, but to do what you suggest, you'd also need a "-min-ply" one.
Not sure if you understood what I was trying to say. The 4-book feature is not yet availabe, that's what I understand from the red. Secondly, no, you don't need the first moves in book 2-4.
Everybody is unique, except me.

User avatar
Ozymandias
Posts: 1071
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Thu Jun 13, 2019 5:46 pm

Rebel wrote:
Thu Jun 13, 2019 3:38 pm
Ozymandias wrote:
Thu Jun 13, 2019 9:22 am
Rebel wrote:
Thu Jun 13, 2019 9:11 am
To bypass the 2Gb limit I made an option in ProDeo to access 4 books but only 1 search is needed.
Book-1 for move 0-5
Book-2 for move 6-10
Book-3 for move 11-15
Book-4 moves >=16

So in theory I can make a book close to 8Gb.
Even if your engine of choice offered the chance to address so many books, which they don't, wouldn't books 2-4 need to include the first moves? There's a "-max-ply" opinion, but to do what you suggest, you'd also need a "-min-ply" one.
Not sure if you understood what I was trying to say. The 4-book feature is not yet availabe, that's what I understand from the red. Secondly, no, you don't need the first moves in book 2-4.
You said "made", which is past tense, so I assumed it was already available; anyway, how do you make a book containing (say) moves 6-10? I tried getting a look at how it works and after executing the program, I get a command line window, with the usual info in the first line, but this message in the second:
tellusererror Polyglot: EngineCommand error: corchess_x64_bmi2.exe
You can't type anything and it closes a short while later.

User avatar
Rebel
Posts: 4463
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot 64-bit

Post by Rebel » Thu Jun 13, 2019 7:15 pm

Ozymandias wrote:
Thu Jun 13, 2019 5:46 pm
Rebel wrote:
Thu Jun 13, 2019 3:38 pm
Ozymandias wrote:
Thu Jun 13, 2019 9:22 am
Rebel wrote:
Thu Jun 13, 2019 9:11 am
To bypass the 2Gb limit I made an option in ProDeo to access 4 books but only 1 search is needed.
Book-1 for move 0-5
Book-2 for move 6-10
Book-3 for move 11-15
Book-4 moves >=16

So in theory I can make a book close to 8Gb.
Even if your engine of choice offered the chance to address so many books, which they don't, wouldn't books 2-4 need to include the first moves? There's a "-max-ply" opinion, but to do what you suggest, you'd also need a "-min-ply" one.
Not sure if you understood what I was trying to say. The 4-book feature is not yet availabe, that's what I understand from the red. Secondly, no, you don't need the first moves in book 2-4.
You said "made", which is past tense, so I assumed it was already available; anyway, how do you make a book containing (say) moves 6-10?
Is also part of what has to be released yet. But it works. See - http://rebel13.nl/download/data.html

It's currently not available from the menu but it's an example that it can be done. A large PGN is split into 3 books. No release because I am not satisfied with the result.
I tried getting a look at how it works and after executing the program, I get a command line window, with the usual info in the first line, but this message in the second:
tellusererror Polyglot: EngineCommand error: corchess_x64_bmi2.exe
You can't type anything and it closes a short while later.
I don't follow. corchess_x64_bmi2.exe ?
Everybody is unique, except me.

User avatar
Ozymandias
Posts: 1071
Joined: Sun Oct 25, 2009 12:30 am

Re: Polyglot 64-bit

Post by Ozymandias » Thu Jun 13, 2019 7:50 pm

Rebel wrote:
Thu Jun 13, 2019 7:15 pm
Is also part of what has to be released yet. But it works. See - http://rebel13.nl/download/data.html

It's currently not available from the menu but it's an example that it can be done. A large PGN is split into 3 books. No release because I am not satisfied with the result.
I tried getting a look at how it works and after executing the program, I get a command line window, with the usual info in the first line, but this message in the second:
tellusererror Polyglot: EngineCommand error: corchess_x64_bmi2.exe
You can't type anything and it closes a short while later.
I don't follow. corchess_x64_bmi2.exe ?
So we're clear, can we make a book containing just moves 6-10, with the released executable? If so, what's the syntax to be used?

The quoted text is a copy/paste of what the program says upon double clicking it. Your guess is as good as mine.

On another machine, it also fails to work, giving the following message:
Can't open file "polyglot.ini": No such file or directory
If an ini file is required, shouldn't it be zipped with the exe?

User avatar
Rebel
Posts: 4463
Joined: Thu Aug 18, 2011 10:04 am

Re: Polyglot 64-bit

Post by Rebel » Fri Jun 14, 2019 7:19 pm

Ozymandias wrote:
Thu Jun 13, 2019 7:50 pm
So we're clear, can we make a book containing just moves 6-10, with the released executable?
Yep, in the next POLY version.
On another machine, it also fails to work, giving the following message:
Can't open file "polyglot.ini": No such file or directory
If an ini file is required, shouldn't it be zipped with the exe?
You don't need a polyglot.ini using --make-book.

For all the other options of Polyglot use (copy) one of the previous versions.
Everybody is unique, except me.

Post Reply