What is faster for endgame tables

[Shaun]

Lars Bremer has done a lot of measurements concerning EGTB on harddisks and
USB flashdrives for our article published in a german computerchess-magazine
in 2006/2007
---> http://www.computerschach.de/index.php? ... Itemid=272

Lars made additional tests with raid-0 in comparison to single drives, please
have a look at:
---> http://www.mustrum.de/artikel/TB-Karussell.pdf

We have done all the measurements with the 5men EGTB.

The blocksize which was requested by the harddisks was mostly 4KB, very seldom
8KB but not even higher ! Lars noticed that with tools that reports the demand
of datas directly on the driver.

Of course one must consider that there are different kinds of flashmemory concerning
the access time. We have used very fast USB sticks, but there are very slow ones on
the market too.

Regards,
G.S.

Gerhard,

if there are any English translations or a summary of the results please can you point me at it.

Thanks

Shaun

[bob]

Hi Pax,
I am using a 8GB usb Memory stick (flash, no minidrive). Acess time very fast (0,7) from transcend - jetflash 160.
All tbs 1-5 men fit on it. The acess is very good. I can see no more low nps with Shredder 9 e.g.

I agree !
Until now there is no HDD and/or technology which is better (faster)
than flashmemory for TBs use, except the Shredderbases in the
main RAM of course.
The only thing which is important is the access time.

Best,
G.S.

that's simply wrong. we read large blocks of data, as we have to decompress a complete block. We need significant transfer rate to pull this off. With a good raid-0 array, your NPS won't drop much at all in almost all cases...

Lars Bremer has done a lot of measurements concerning EGTB on harddisks and
USB flashdrives for our article published in a german computerchess-magazine
in 2006/2007
---> http://www.computerschach.de/index.php? ... Itemid=272

Lars made additional tests with raid-0 in comparison to single drives, please
have a look at:
---> http://www.mustrum.de/artikel/TB-Karussell.pdf

We have done all the measurements with the 5men EGTB.

The blocksize which was requested by the harddisks was mostly 4KB, very seldom
8KB but not even higher ! Lars noticed that with tools that reports the demand
of datas directly on the driver.

Of course one must consider that there are different kinds of flashmemory concerning
the access time. We have used very fast USB sticks, but there are very slow ones on
the market too.

Regards,
G.S.

Here's the thing. I did the original testing for compressed tablebases, and tried all sorts of blocksizes to see which offered the best tradeoff with respect to CPU operations vs disk drive transfers. We are not using 4K blocksizes. I honestly don't remember whether we settled on 8K or 16K for the blocksize, Eugene could chime in there. But we definitely read more than 4K of data.

I will add that all my testing was done exclusively on Linux boxes, using the fastest SCSI drives available at the time (10K U320). I now use 15K U320 drives.

I have not tried every sort of flash drive, SSD, and so forth, but so far I have not found a thing that is better than the raid-0 array I currently use...

[ThatsIt]

Hi Shaun !

Gerhard,

if there are any English translations or a summary of the results please can you point me at it.

Thanks
Shaun

Unfortunately not.
I will ask Lars Bremer, perhabs he is able to translate.
My english is not good enough.

Best to you !
G.S.

[ThatsIt]

Hi !

Here's the thing. I did the original testing for compressed tablebases, and tried all sorts of blocksizes to see which offered the best tradeoff with respect to CPU operations vs disk drive transfers. We are not using 4K blocksizes. I honestly don't remember whether we settled on 8K or 16K for the blocksize, Eugene could chime in there. But we definitely read more than 4K of data.
I will add that all my testing was done exclusively on Linux boxes, using the fastest SCSI drives available at the time (10K U320). I now use 15K U320 drives.
I have not tried every sort of flash drive, SSD, and so forth, but so far I have not found a thing that is better than the raid-0 array I currently use...

Thanks for the explanation.
Perhabs it would be interesting for some users if you compare
your scsi-system with some fast usb flash if possible ?

Regards,
G.S.

[smirobth]

I had always thought that flash was too slow for tablebases, but all the discusion here convinced me to try it. The results of a fast (200x "adata 8GB "my flash") flash memory stick versus a SATA 7200 RPM hard drive are definately good for the flash. It would be nice to see results versus a 15,000 RPM SCSI. I may dig mine out of the garage out of curiosity. In the mean time here are the results I got versus the SATA 7200 RPM with 2 MB tablebase cache, 128MB hashtables, Fritz 10 "tablebase depth"=0 (to maximize tablebase hits), all 3-4-5 man and no 6 man tablebases. In each line the first time is for the flash stick and the second (always slower) time is with the SATA 7200 RPM hard drive:

Morozevich,A-Mamedyarov,S, 2006
8/4K3/3P4/1N5p/6kp/6P1/8/8 b - -
Depth: 7/23 170kN, tb=540 0:00:00 0:00:01
Depth: 8/29 657kN, tb=3632 0:00:00 0:00:06
Depth: 9/29 1264kN, tb=7381 0:00:01 0:00:09
Depth: 10/26 1489kN, tb=8914 0:00:01 0:00:12
Depth: 11/25 1546kN, tb=9292 0:00:01 0:00:12
Depth: 12/30 2252kN, tb=15402 0:00:02 0:00:18
Depth: 13/31 2915kN, tb=21609 0:00:03 0:00:23
Depth: 14/32 5284kN, tb=43141 0:00:06 0:00:33
Depth: 15/35 18346kN, tb=182711 0:00:20 0:01:05
Depth: 16/36 49398kN, tb=530337 0:00:53 0:02:03
Depth: 17/42 105mN, tb=1224023 0:01:53 0:03:20

Kasparov,G-Kramnik,V, 1994
8/6k1/8/2K2p1P/1P3n2/5B2/8/8 b - -
Depth: 9/15 23kN, tb=330 0:00:00 0:00:00
Depth: 10/17 34kN, tb=456 0:00:00 0:00:01
Depth: 11/18 59kN, tb=827 0:00:00 0:00:02
Depth: 12/20 111kN, tb=1452 0:00:00 0:00:03
Depth: 13/30 367kN, tb=5240 0:00:02 0:00:07
Depth: 14/25 536kN, tb=7581 0:00:02 0:00:09
Depth: 15/32 1108kN, tb=17667 0:00:05 0:00:16
Depth: 16/33 2118kN, tb=32993 0:00:07 0:00:24
Depth: 17/28 3359kN, tb=48823 0:00:10 0:00:31
Depth: 18/33 5346kN, tb=74601 0:00:14 0:00:40
Depth: 19/37 12110kN, tb=183224 0:00:26 0:01:07
Depth: 20/35 17523kN, tb=258396 0:00:34 0:01:22
Depth: 21/36 26255kN, tb=373442 0:00:46 0:01:43
Depth: 22/42 48702kN, tb=719882 0:01:18 0:02:38
Depth: 23/40 78722kN, tb=1110124 0:01:54 0:03:31

Adams,M-Gelfand,B, 2006
4r3/2k5/2P3N1/3K1P2/8/8/8/8 w - -
Depth: 8/1 95kN, tb=2081 0:00:00 0:00:02
Depth: 9/21 113kN, tb=2509 0:00:00 0:00:02
Depth: 10/24 155kN, tb=3540 0:00:00 0:00:02
Depth: 11/20 179kN, tb=4235 0:00:01 0:00:03
Depth: 12/22 255kN, tb=6247 0:00:01 0:00:03
Depth: 13/22 369kN, tb=9405 0:00:01 0:00:04
Depth: 14/31 605kN, tb=16008 0:00:02 0:00:06
Depth: 15/32 1157kN, tb=31059 0:00:03 0:00:09
Depth: 15/41 2291kN, tb=60107 0:00:05 0:00:13
Depth: 16/68 4102kN, tb=109563 0:00:08 0:00:20
Depth: 17/45 6322kN, tb=172556 0:00:11 0:00:25
Depth: 18/45 8600kN, tb=241286 0:00:15 0:00:32
Depth: 19/45 10577kN, tb=303013 0:00:18 0:00:36
Depth: 20/46 23562kN, tb=696976 0:00:36 0:01:04
Depth: 21/58 37338kN, tb=1112556 0:00:52 0:01:25
Depth: 21/58 46999kN, tb=1380196 0:01:00 0:01:34
Depth: 21/58 57288kN, tb=1673621 0:01:08 0:01:42
Depth: 21/97 137mN, tb=3238198 0:02:22 0:03:23

Karpov,A-Anand,V, 1998
7q/8/1P2R3/4R3/7K/7P/5k2/8 w - -
Depth: 8/31 117kN, tb=287 0:00:00 0:00:01
Depth: 9/29 361kN, tb=1143 0:00:01 0:00:03
Depth: 9/30 529kN, tb=1749 0:00:01 0:00:04
Depth: 9/1 897kN, tb=2664 0:00:01 0:00:05
Depth: 9/1 1002kN, tb=2938 0:00:02 0:00:05
Depth: 9/1 1045kN, tb=3102 0:00:02 0:00:05
Depth: 9/1 1071kN, tb=3261 0:00:02 0:00:05
Depth: 10/31 1227kN, tb=3959 0:00:02 0:00:06
Depth: 11/1 1527kN, tb=5081 0:00:02 0:00:07
Depth: 12/1 3167kN, tb=11774 0:00:05 0:00:13
Depth: 13/36 6045kN, tb=26443 0:00:08 0:00:21
Depth: 14/44 16776kN, tb=79116 0:00:19 0:00:40
Depth: 15/53 54457kN, tb=334503 0:00:56 0:01:34
Depth: 16/54 76074kN, tb=488671 0:01:17 0:02:02
Depth: 16/76 377mN, tb=2832102 0:05:46 0:06:58

[Spock]

Yes - flash drives are a lot faster than they used to be

And remember, VISTA now has it's ReadyBoost feature, where you put in a flash drive and Vista uses that to page memory in and out instead of the hard drive, because it is faster than the hard drive

That is pretty compelling evidence that the current generation of fast flash drives could be quicker than hard drives for tablebases

And the big bonus ?? Flash drives aren't mechanical and shouldn't fail like a hard-worked hard drive

Just my opinion....

[ThatsIt]

Hi !

I had always thought that flash was too slow for tablebases, but all the discusion here convinced me to try it. The results of a fast (200x "adata 8GB "my flash") flash memory stick versus a SATA 7200 RPM hard drive are definately good for the flash. It would be nice to see results versus a 15,000 RPM SCSI. I may dig mine out of the garage out of curiosity. In the mean time here are the results I got versus the SATA 7200 RPM with 2 MB tablebase cache, 128MB hashtables, Fritz 10 "tablebase depth"=0 (to maximize tablebase hits), all 3-4-5 man and no 6 man tablebases. In each line the first time is for the flash stick and the second (always slower) time is with the SATA 7200 RPM hard drive:

Hi !

Would you be so kind to make a measure with your stick ?
It would be nice to find out how fast the stick really is.

Please use the tool "H2benchw" from here:
[ftp://ftp.heise.de/pub/ct/ctsi/h2benchw.zip]

and copy it into a folder of your choice.

First you have to find out which device number your usb-stick has.

You can use the devicemanager to find out or use h2benchw.
Type (in a console-window)
h2benchw 0 -english
or
h2benchw 1 -english
or
h2benchw 2 -english
...
to find out, what number your stick has.

After that type
h2benchw 1 -s -z -english
(if the usb stick is device number 1 on your system)

You will get an output like this:
(example in german)
H2bench -- by Harald Bögeholz & Lars Bremer / c‘t Magazin für
Computertechnik
Version 3.6/Win32, Copyright (C) 2002 Heise Zeitschriften Verlag GmbH & Co. KG
Niederländische Übersetzung: F&L Technical Publications B.V.
!!! WARNUNG: Ergebnisse werden nicht abgespeichert!
Kapazität: 7920045 Sektoren=3867 MByte, CHS=(493/255/63)
10 Sekunden Timerüberprüfung (Win32) ............. Ok.
Timerauflösung: 0.279 µs, 3.580 MHz
Timerstatistik: 2653483 Aufrufe, min 1.68 µs, mittel 1.85 µs, max 228.24 µs
Einige Sektoren lesen zum Aufwärmen... Fertig.
Zonenmessung Lesen: Kalibrierung... ca. 18.2 MByte/s bei 50% der Kapazität.
Lese mit 847 Messpunkten (73 Blöcke … 128 Sektoren = 4.56 MByte)
Geschätzte Laufzeit: 4 Minuten...Fertig.
Dauertransferrate Lesen: Mittel 18811.7, Min 17041.1, Max 19009.9 [KByte/s]
Messung der mittleren Zugriffszeit (gesamte Platte):
Lesen... 0.99 ms (Min. 0.36 ms, Max. 4.45 ms)
Zugriffszeit innerhalb der ersten 504 MByte
Lesen... 0.74 ms (Min. 0.48 ms, Max. 4.00 ms)


The bench test will last about 3-5 minutes.

"Lesen" is the crucial part. This example shows values 0.99 (for the
whole capacity of the stick) and 0.74ms (for the first 504MB).
Very good (fast) is 0.50-0.99, good enough for EGTB use is 1.00-1.30.

Best wishes and THX,
G.S.

[smirobth]

Hi !

I had always thought that flash was too slow for tablebases, but all the discusion here convinced me to try it. The results of a fast (200x "adata 8GB "my flash") flash memory stick versus a SATA 7200 RPM hard drive are definately good for the flash. It would be nice to see results versus a 15,000 RPM SCSI. I may dig mine out of the garage out of curiosity. In the mean time here are the results I got versus the SATA 7200 RPM with 2 MB tablebase cache, 128MB hashtables, Fritz 10 "tablebase depth"=0 (to maximize tablebase hits), all 3-4-5 man and no 6 man tablebases. In each line the first time is for the flash stick and the second (always slower) time is with the SATA 7200 RPM hard drive:

Hi !

Would you be so kind to make a measure with your stick ?
It would be nice to find out how fast the stick really is.

Hi,

It seems that my average read time for this stick is 0.67mS. Here is the output I got (I used the -english option):

Capacity: CHS=(1011/255/63), 16241715 sectors = 7931 MByte

Sustained transfer rate (block size: 128 sectors):
Reading: average 20222.1, min 19615.7, max 21249.8 [KByte/s]

Random access read: average 0.67, min 0.49, max 0.88 [ms]
Random access read (<504 MByte): average 0.51, min 0.44, max 0.71 [ms]
Detailed results:
-----------------

Zone measurement read [KByte/s]:
<snip>

[gigabyte137]

End in end from all reading what is conclusion? Raid 0 or flash drives. keep in mind I have 300 gigs of 6 men tables but what about for 1-5 man tables?

[smirobth]

End in end from all reading what is conclusion? Raid 0 or flash drives. keep in mind I have 300 gigs of 6 men tables but what about for 1-5 man tables?

I used to think a 15,000 RPM U320 SCSI RAID 0 would be fastest. I have now changed my mind, and while I'm still not 100% certain I suspect that a FAST flash will be at least as good and probably even better. The flash will definitely be smaller, cheaper, quieter and take less power. However you won't be able to get 300 gigs on a flash so if you want max speed for your 6-man tables you'll still need a 15,000 RPM hard drive, and RAID 0 would help too.

Once you have forked over the big bucks for a 15,000 RPM U320 RAID 0 you can afford the marginal extra cost (~$100) for a fast flash stick and then you can test the difference and tell us.