Tablebases loaded in System Memory

[FlavusSnow]

A while back Robert Houdart recommended loading critical tablebase data into memory as a WDL file.

Has anyone ever tried mounting a temporary file system in system memory, copying the tablebases to this temporary system, then using this as the default path for the engine's tablebases?

3-4-5 man TBs add up to something like 6.5 GB so if you mounted 8 GB of ram for that plus another gig for the engine's main hash, it'd only take 12 gig or so to be comfortably running this all in ram. Ram is cheap these days so this seems practical.

Any comments? What kind of performance gain could be gotten from doing this? I'd doubt anything truly noticeable...

[sje]

What kind of performance gain could be gotten from doing this? I'd doubt anything truly noticeable...

It all depends.

1) It depends on how frequently the tablebases are accessed. If accessed only a few dozen times per search, then RAM vs disk makes no noticeable difference.

2) It depends on which tablebases are present as the N-man classes, if a (nearly) complete set are present, shadow all classes of length M < N.

3) It depends on the compression techniques in use. Uncompressed tablebases are always faster, but will take more space than what might be available in RAM.

4) It depends on the opportunity cost of RAM which isn't available for use as transposition table space.

A good approach is to instead use the RAM for more transposition storage and get an SSD (solid state drive) for storing tablebases.

[sje]

A performance estimate for tablebase access using an SSD:

For the following 120 GB SSD: http://eshop.macsales.com/item/OWC/SSDMX120/

The reported 4 KB block random read throughput rate is 26.4 MB/second with a fast SATA interface. That's about 6,800 reads/second, so 6,800 tablebase accesses/second. This is claimed to be 47 times faster than a 5,400 RPM hard drive. The model referenced above priced at US$183 can contain all the uncompressed 2-5 tablebase files (68 GB total) with room to spare.

The exact rate will no doubt vary depending on processor performance as there's a lot of overhead for each system I/O call.

I'll probably get one of these after the perft(13) run completes and use it for the tablebases. I won't use it for the opening book because the book, only a few dozen MB, easily fits into RAM and loads from a regular drive in under a second.

[Edmund]

A while back Robert Houdart recommended loading critical tablebase data into memory as a WDL file. [...]
3-4-5 man TBs add up to something like 6.5 GB so if you mounted 8 GB of ram for that plus another gig for the engine's main hash, it'd only take 12 gig or so to be comfortably running this all in ram. [...]

compressed 5-men bitbases consume roughly 200mb (Scorpio bitbases)

[Daniel Shawul]

One of these days I am gonna have to find time for generating 6 men bitbases and see what size they will be after being compressed. people shoudnt be using gigantic tbs in the 21st century : ) cheers

[bob]

One of these days I am gonna have to find time for generating 6 men bitbases and see what size they will be after being compressed. people shoudnt be using gigantic tbs in the 21st century : ) cheers

If you don't use 'em you are going to draw a lot of won games...

[Edmund]

One of these days I am gonna have to find time for generating 6 men bitbases and see what size they will be after being compressed. people shoudnt be using gigantic tbs in the 21st century : ) cheers

If you don't use 'em you are going to draw a lot of won games...

using them in search is not necessary and thats the critical part. Once you reached the won position even an online-lookup would suffice. Alternatively also dropping parts of the dtc-tablebase has been discussed before and solving the missing information through search.