A possible way to speed up tablebases

[Norm Pollock]

First off, I do not use an SSD which is what you need for maximum efficiency with tablebases. However my idea will apply to those who do.

I use a USB 3.0 stick for syzygy 3-4-5-6-man tbs. I have a 3.0 usb port. Not quite as good as an SSD, but by setting probe depth to 12, I feel I am in good shape.

My machine has 4 cores with hyperthreading, resulting in 8 threads. I tried using 2 USB 3.0 sticks with 3-4-5-man tbs on one, and 6-man on the other.

While one thread is using 6-man, another thread could be using the 3-4-5-man. This would make the queue move faster. Sort of like having an express line at the supermarket.

Likewise, for those with an SSD, use 2 SSD's and split off the 3-4-5 from the 6-man.

While I cannot measure any improvement in speed, I do notice both USB 3.0 sticks flashing at the same time, indicating that there are two queues.

[yurikvelo]

I tried using 2 USB 3.0 sticks with 3-4-5-man tbs on one, and 6-man on the other.

While one thread is using 6-man, another thread could be using the 3-4-5-man. This would make the queue move faster.

AFAIK, N-1-men bases never get probed, if you have N-men bases (at least before you have +128 eval and engine start to resolve DTM score)

Better divide 6-men *.rtbw files in 2 subsets on 2 sticks.

5-men are so small - can be kept in RAM-disk, but there is no point. Free as much RAM as possible, so OS will use it for I/O buffers reading your 6-men *.rtbw files

[Evert]

I don't think that works. As far as I know disk I/O all goes through a single thread at some point, so if you try to access the disk from multiple threads they end up waiting for eachother.

It's a few years ago that I ran into this though, so things may have changed. It may also depend on the kernel options (in Linux, I don't think you can configure those in Windows).

[syzygy]

I tried using 2 USB 3.0 sticks with 3-4-5-man tbs on one, and 6-man on the other.

While one thread is using 6-man, another thread could be using the 3-4-5-man. This would make the queue move faster.

AFAIK, N-1-men bases never get probed, if you have N-men bases (at least before you have +128 eval and engine start to resolve DTM score)

The N-1-men will be probed while the N-men are getting probed because of how the compression scheme works. This why N-men TBs absolutely do not work if N-1-men are absent.

The N-1-men are so small that they will quickly be cached in RAM by the OS. (And indeed a RAM disk is a waste.)

[syzygy]

I don't think that works. As far as I know disk I/O all goes through a single thread at some point, so if you try to access the disk from multiple threads they end up waiting for eachother.

I/O requests to different devices will not be serialised. And I/O requests to the same device can be reordered by both Windows and Linux where that makes sense.

See e.g. https://en.wikipedia.org/wiki/I/O_scheduling