Stockfish version with hash saving capability

[Zerbinati]

You are right Rodolfo
However I decided to delete repository

I don't need to be right, Marco. I only wish both you and Joerg come to an agreement about:

a) including your changes, or
b) making your own SF fork.

In both cases, what you're doing is greatly interesting.

No problems Rodolfo .. I'm not bothered
But I prefer to avoid unnecessary controversy
So many compilations are released
And obviously code and stockfish + numerous patches that improve or offer interesting additional parameters
All this is possible even without creating a repositary.

[Rodolfo Leoni]

You are right Rodolfo
However I decided to delete repository

I don't need to be right, Marco. I only wish both you and Joerg come to an agreement about:

a) including your changes, or
b) making your own SF fork.

In both cases, what you're doing is greatly interesting.

No problems Rodolfo .. I'm not bothered
But I prefer to avoid unnecessary controversy
So many compilations are released
And obviously code and stockfish + numerous patches that improve or offer interesting additional parameters
All this is possible even without creating a repositary.

Look for a PM...

[yorkman]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

[Zerbinati]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

Here is Daniel Jose code
https://github.com/Zerbinati/Stockfish/ ... 6d9b02515b

Here's the code Daniel corrected to fix the Hash 2gb limit

Code: Select all

bool TranspositionTable::save() { 
   std::ofstream b_stream(hashfilename, 
      std::fstream::out | std::fstream::binary); 
   if (b_stream) 
   { 
      //b_stream.write&#40;reinterpret_cast<char const *>&#40;table&#41;, clusterCount * sizeof&#40;Cluster&#41;); 
      for &#40;long long i = 0; i < clusterCount * sizeof&#40;Cluster&#41;; i += &#40;1 << 30&#41;) &#123; //1GB 
         long long j = __min&#40;&#40;1 << 30&#41;, &#40;clusterCount * sizeof&#40;Cluster&#41;) - i&#41;; 
         b_stream.write&#40;reinterpret_cast<char const *>&#40;table&#41; + i, j&#41;; 
      &#125; 
      return &#40;b_stream.good&#40;)); 
   &#125; 
   return false; 
&#125; 

void TranspositionTable&#58;&#58;load&#40;) &#123; 
   //file size&#58; https&#58;//stackoverflow.com/questions/2409504/using-c-filestreams-fstream-how-can-you-determine-the-size-of-a-file
   std&#58;&#58;ifstream file; 
   file.open&#40;hashfilename, std&#58;&#58;ios&#58;&#58;in | std&#58;&#58;ios&#58;&#58;binary&#41;; 
   file.ignore&#40;std&#58;&#58;numeric_limits<std&#58;&#58;streamsize>&#58;&#58;max&#40;)); 
   std&#58;&#58;streamsize size = file.gcount&#40;); 
   file.clear&#40;);   //  Since ignore will have set eof. 
   resize&#40;size / 1024 / 1024&#41;; 
   file.seekg&#40;0, std&#58;&#58;ios&#58;&#58;beg&#41;; 
   file.read&#40;reinterpret_cast<char *>&#40;table&#41;, clusterCount * sizeof&#40;Cluster&#41;); 
&#125;

[cdani]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

Hello. Have you tried with the latest version I published few days ago? Should be solved.

Andscacs:

www.andscacs.com/downloads/andscacs091237.zip

Stockfish:

www.andscacs.com/downloads/stockfish_x6 ... vehash.zip

[giovanni]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

Hello. Have you tried with the latest version I published few days ago? Should be solved.

Andscacs:

www.andscacs.com/downloads/andscacs091237.zip

Stockfish:

www.andscacs.com/downloads/stockfish_x6 ... vehash.zip

Hi Jose'. Thanks again for your efforts. Can we assume now that the your code in Stockfish gave to the software, as long as persistent analysis is concerned, all the capabilities that were in Stockfish-TCEC6-32-PA_GT?

[Dann Corbit]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

Hello. Have you tried with the latest version I published few days ago? Should be solved.

Andscacs:

www.andscacs.com/downloads/andscacs091237.zip

Stockfish:

www.andscacs.com/downloads/stockfish_x6 ... vehash.zip

Hi Jose'. Thanks again for your efforts. Can we assume now that the your code in Stockfish gave to the software, as long as persistent analysis is concerned, all the capabilities that were in Stockfish-TCEC6-32-PA_GT?

Stockfish PA GTB could read a giant file of analyzed EPD records and both write the data to the permanent hash file on disk and also seed the live hash table with it.

[Rodolfo Leoni]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

Hello. Have you tried with the latest version I published few days ago? Should be solved.

Andscacs:

www.andscacs.com/downloads/andscacs091237.zip

Stockfish:

www.andscacs.com/downloads/stockfish_x6 ... vehash.zip

Hi Jose'. Thanks again for your efforts. Can we assume now that the your code in Stockfish gave to the software, as long as persistent analysis is concerned, all the capabilities that were in Stockfish-TCEC6-32-PA_GT?

Stockfish PA GTB could read a giant file of analyzed EPD records and both write the data to the permanent hash file on disk and also seed the live hash table with it.

Systems are different. Stockfish PA GTB uses an additional hash to store data. It grants nothing get lost while going on with analysis. The hashes can be used for normal games too, and engine can gain some "experience" from playing the same position repeatly. The option "importepd <file.epd>" gives engine the capability to use previous saved data for analysis. While thinking on a position, PV entries are automatically saved into file (according with depth setting).

The "never clear hash" option of Daniel's Stockfish_savehash afftecs the main hash, only in infinite analysis mode. The whole search of main hash becomes "persistent", until more valuable entries overwrite less valuable ones. That means analysis results could be lost sooner or later, if you use the hash file a lot of times. In my case, I'm using it in two correspondence tournaments. I believe I lost some data since the first time I used a file, just because I'm using one file per game, and some games are already in early ending. I assume, then, most early middlegame data got lost.

Daniel's system (based on HGM idea) has an advantage: there's no disk access while on analysis. It needs only some time to load/save hashes.

[Leo]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

Hello. Have you tried with the latest version I published few days ago? Should be solved.

Andscacs:

www.andscacs.com/downloads/andscacs091237.zip

Stockfish:

www.andscacs.com/downloads/stockfish_x6 ... vehash.zip

Hi Jose'. Thanks again for your efforts. Can we assume now that the your code in Stockfish gave to the software, as long as persistent analysis is concerned, all the capabilities that were in Stockfish-TCEC6-32-PA_GT?

No way.

[Dann Corbit]

First, thanks for this overdue feature. I've been waiting for SF to have this option for a very long time now.

Second, unfortunately the code isn't working properly. It only works up to a 2 GB hash size and it takes extremely long to save it...about 10 mins. For comparison, I save the asmFish hash (32 GB) in about 7 mins).

Hopefully you can fix that...or if you can at least provide instructions on how to add this feature into the source code ourselves (including a fix) and which file we're to add it in we'd really appreciate it.

Hello. Have you tried with the latest version I published few days ago? Should be solved.

Andscacs:

www.andscacs.com/downloads/andscacs091237.zip

Stockfish:

www.andscacs.com/downloads/stockfish_x6 ... vehash.zip

Hi Jose'. Thanks again for your efforts. Can we assume now that the your code in Stockfish gave to the software, as long as persistent analysis is concerned, all the capabilities that were in Stockfish-TCEC6-32-PA_GT?

No way.

Not much is missing for a better start.
Daniel is very smart (not to be confused with Danniel who is dumb as a box of ball peen hammers). Loading the hash from a big file of EPD is all that is missing. And SF already has a function to parse EPD.