SSD for table lookup

[Don]

Solid State disks are becoming faster and larger. How can a computer chess program take advantage of this?

The primary benefit is extremely low read latency times.

The obvious thing is to use this for storing endgame databases. Who is trying this and how much does the program benefit over storing this on a physical drive?

[Gian-Carlo Pascutto]

Many people are using USB drives for this for the exact same reason. The main benefit is that the program slows down less when accessing tb heavily.

[diep]

Solid State disks are becoming faster and larger. How can a computer chess program take advantage of this?

The primary benefit is extremely low read latency times.

The obvious thing is to use this for storing endgame databases. Who is trying this and how much does the program benefit over storing this on a physical drive?

Not easy to accomplish but much better is supercompression of the 3-6 men and put them in a gigabyte of RAM.

No matter the latency in milliseconds, all those sticks, drives and SSD type $20k solutions are just too slow compared to simple RAM.

Well i make 1 mistake, i've got some dudes spamming me about some big disks that are really expensive and have some hundreds of nanoseconds access time. Yet your car might be cheaper than that

Note i managed to get some 6 men into a few hundreds of kilobytes when i experimented with it a few years ago. Yet that decompression code is too slow and it decompresses entire thing instead of giving O ( log n ) access to it. Such compression systems do not exist simply yet, as no one wants to pay for them.

With fast GPU's being able to realtime decompress things maybe it will exist some day. CPU power is near to unlimited now compared to price of low latency storage.

Vincent

[diep]

Many people are using USB drives for this for the exact same reason. The main benefit is that the program slows down less when accessing tb heavily.

USB is central locking your file system.

Vincent

*you might not care about that during tournaments, but i bet you do when performing daily work
**some side effect may cause effectively that you cannot use your file system - i don't care for what causes it, but it just locks out the entire fs and if your raid10 system gets locked out like that that is a pain in the **********

[ernest]

supercompression of the 3-6 men and put them in a gigabyte of RAM.
Note i managed to get some 6 men into a few hundreds of kilobytes when i experimented with it a few years ago.

Hi Vincent,
1200 GB (already in compressed form) reduced to 1 GB!?
That seems quite a feat!!!
Do you have a link for that?

[Vinvin]

5 pieces : Nalimov=7500 MB ; Shredderbase=157 MB
Ratio = about 50
6 pieces : Nalimov=1200 GB ; -> compressible to 1200 GB/50= 24 GB ?
I hope so ...

supercompression of the 3-6 men and put them in a gigabyte of RAM.
Note i managed to get some 6 men into a few hundreds of kilobytes when i experimented with it a few years ago.

Hi Vincent,
1200 GB (already in compressed form) reduced to 1 GB!?
That seems quite a feat!!!
Do you have a link for that?

[ernest]

5 pieces : Nalimov=7500 MB ; Shredderbase=157 MB
Ratio = about 50

Salut Vinvin,
Yes, but Nalimov is... Nalimov (tablebases, with DTM), while Shredderbase is BitBase (only W/L/D info) so you cannot use it as ratio for future tablebase compression.

[michiguel]

Solid State disks are becoming faster and larger. How can a computer chess program take advantage of this?

The primary benefit is extremely low read latency times.

The obvious thing is to use this for storing endgame databases. Who is trying this and how much does the program benefit over storing this on a physical drive?

I am trying this, but I have not bought the SSDs yet
My own tablebases are not compressed and will benefit from SSDs. Originally, my plan was to develop a compression system too, but when I tried them "as is" they were fast enough already with the cache I developed (I just do not probe in the last two plies). Using SSDs, there is no reason for compressing the 5pc tablebases. They fit already and the latency will be ok with no overhead from uncompressing on the fly.

So, this is speculation because I have not tried a SSD, but I guess they will be very good for uncompressed TBs.

Miguel

[diep]

5 pieces : Nalimov=7500 MB ; Shredderbase=157 MB
Ratio = about 50

Salut Vinvin,
Yes, but Nalimov is... Nalimov (tablebases, with DTM), while Shredderbase is BitBase (only W/L/D info) so you cannot use it as ratio for future tablebase compression.

Oh comeon, DTM only got used because they wanted to brag.

Now Nalimov, let's be clear i do not want to say he didn't do a lot of effort, as he did. He did do it and did do it within the given constraints very well (assuming you don't mind bizarre template definitions, that increase your executable size by half a megabyte). Never spit on someone who did do the hard job before you. There was thompson, Edwards and then Nalimov.

Now it is the 21th century and we want to improve upon it to something realistic. Using WDL is not new, already gets used for as long as i live nearly.

Many engines already are using WDL for some years now in chess. Already in 90s Mathias Feist told me he really wanted to switch to WDL for Fritz as well. Of course reality is you just know the name rybka and shredder and that's about it, but there is really a lot of engines out there, and really a lot of them are using WDL.

In checkers, both the american variation and 10x10 international checkers, they *always* only used win/draw/loss scores and never DTM nor any of the other exotic forms such as DTC that initially was getting used.

The compression ratio between DTM 1300GB and a gig, or when you do it sloppy some 20-40 gigs (see shredder), the difference is too huge to ignore.

The difficult task is a customized generic compression that is really good and up to todays standards. Compression is a science in itself.

From the oldie 80s and start 90s algorithms, Kadatch is doing a good job. Most use him. However when i test on my EGTBs, better forms of compression (so generic compression) are up to factor 4 better.

Hear me, FACTOR 4.

Note that Compression for Nalimov's has different characteristics than for WDL. WDL allows way more n-th order dimensions (google for PPM). The bigger the EGTBs, the more you can achieve there.

However you must keep in mind that uncompressed in Diep format, the 6 men are 1.05 TB

that's already a lot tinier than the uncompressed version of Nalimov, which are i guess at least 5 times bigger.

Ignoring a factor 5 would be impossible, let alone a factor 100 to 1000.

Even default compression at diep's 6 men, they are 34GB compressed. So that is without special treatment yet, which we all know is possible.

Every EGTB i did do some effort for, has a compression ratio of factor 5000 effectively or more.

There is no hard 'perfect' solution for the problem of course.

It is always possible to do better. If you consider how much time EGTBs all together have eaten from me (writing 3 different generators, 4 different formats to store them in, joining the effort to spread them and so on), then the amount of elopoints EGTBs win in todays computer chess is pretty little, whereas the effect of them at checkers and 10x10 checkers is pretty devastating. Maybe it will come in chess also, who knows?

I do not know. I wouldn't dare to predict it to be honest.

Vincent

[ernest]

OK Vincent, so you are speaking about WDL bases (BitBases, ShredderBases, ...) and a better compression.

I don't want to reopen that big debate, but nobody seems to have a clear idea about how to converge to Mate with WDL bases alone.