What is faster for endgame tables

[gigabyte137]

I remember seeing it before but I can't find it.... Raid 0 (2) Seagate 320Gb drives or some kind of flashdrive or usb jumpdrive? Overall performance which is better for endgame tables and only endgame tables? Thanks...

[bob]

I remember seeing it before but I can't find it.... Raid 0 (2) Seagate 320Gb drives or some kind of flashdrive or usb jumpdrive? Overall performance which is better for endgame tables and only endgame tables? Thanks...

raid-0 by a _large_ margin...

[LaurenceChen]

Raid 0 has a huge drawback, if one of the hard drives die, you lose all your data.
The best choice would be RAID 5 with 5 hard drives, 6 would be ideal. This will give you a similar performance of a RAID 0, or at least a close performance to RAID 0 without the worry of losing data when one of the hard drives die out.

[bob]

Raid 0 has a huge drawback, if one of the hard drives die, you lose all your data.
The best choice would be RAID 5 with 5 hard drives, 6 would be ideal. This will give you a similar performance of a RAID 0, or at least a close performance to RAID 0 without the worry of losing data when one of the hard drives die out.

raid-5 is not as fast, particularly if it is software raid. Lots of computation to compute the redundancy stuff. Whereas software raid-0 works well using SCSI disks (not IDE/SATA) that can read/write in parallel with each other. Hardware raid-5 is better than software raid-5 from a speed perspective, but also pricier...

I don't worry about losing the data since it doesn't change. Any backup media will work well and the redundancy is not needed from that perspective.

[M ANSARI]

RAID 0 array will not perform as fast as an solid state drive. The problem is that solid state drives need to have a live feed or their batteries will die out. Jump drives are probably the next best. The problem is the limited amount of useful capacity. You could probably get 64GB for around $500 ... but when 6 man EGTB are 1.2 TB ... 2 seagate 750 GB perpendicular drives in a RAID 0 array might be a better choice. Maybe a combination of both is the best bet.

I cannot imagine that there is not a better way to compress EGTB footprint. A lot of engines perform much worse if they have to access EGTB's due to that fact. Maybe there is a way to first assess if a certain ending is worth searching in an EGTB or not. I am not sure why it is that EGTB's do not use compression ... maybe they are already compressed to the max. I did hear that Shredder uses some sort of proprietary EGTB's but am not sure how that works. At the moment all engine writers seem to prefer to just use off the shelf code that allows Nalimov EGTB usage. I wonder if this can be improved upon.

[smirobth]

RAID 0 array will not perform as fast as an solid state drive. The problem is that solid state drives need to have a live feed or their batteries will die out. Jump drives are probably the next best. The problem is the limited amount of useful capacity. You could probably get 64GB for around $500 ... but when 6 man EGTB are 1.2 TB ... 2 seagate 750 GB perpendicular drives in a RAID 0 array might be a better choice. Maybe a combination of both is the best bet.

I am not so sure that a solid state drive is necessarily faster than raid 0; if you were to compare their speed to a RAID 0 (4 disk) array of Ultra 320 SCSI 15,000 RPM drives I think the RAID 0 will often be faster. Besides, as you mention, there is no practical/cost effective way to even get a solid state drive that is sufficiently large enough for 6 man tablebases. The solid state drive would only be good for 5 man or fewer, which means still needing hard drives.

And a flash based jump drive will not even be as fast as a run of the mill SATA 7200 RPM HDD (and will have the same limited storage capacity as the solid state drives).

I think for tablebase applications some type of hard drive configuration will be best. And if speed is the only concern, a RAID 0 array of 15,000 RPM Ultra 320 SCSI's will be hard to beat. It will also be noisy, power hungry, noisy, generate a lot of heat, noisy and you'll want to have the data backed up elsewhere. And did I mention NOISY? But it will be fast.

[bob]

RAID 0 array will not perform as fast as an solid state drivequote]

That is in theory. In reality raid0 murders solid state drives, because they are all using things like FLASH memory which is ridiculously slow. Yes, if you had a large DRAM disk (Cray Research builds them but they are not cheap) then they would be fast. But if you re-read the original post, he mentioned things like "thumbdrives" and the like and they are horribly slow compared to an array of SCSI drives.

.[/ The problem is that solid state drives need to have a live feed or their batteries will die out. Jump drives are probably the next best. The problem is the limited amount of useful capacity. You could probably get 64GB for around $500 ... but when 6 man EGTB are 1.2 TB ... 2 seagate 750 GB perpendicular drives in a RAID 0 array might be a better choice. Maybe a combination of both is the best bet.

I cannot imagine that there is not a better way to compress EGTB footprint. A lot of engines perform much worse if they have to access EGTB's due to that fact. Maybe there is a way to first assess if a certain ending is worth searching in an EGTB or not. I am not sure why it is that EGTB's do not use compression ... maybe they are already compressed to the max. I did hear that Shredder uses some sort of proprietary EGTB's but am not sure how that works. At the moment all engine writers seem to prefer to just use off the shelf code that allows Nalimov EGTB usage. I wonder if this can be improved upon.

EGTBs are already compressed, using a compression technique that is highly optimized specifically for EGTBs and the way the data is stored... They can be made smaller, but then they would not be usable inside the search. As it is now, they are compressed into "chunks" and each chunk has to be decompressed to access data within it. Who'd want to do a single probe and have to decompress a multi-gigiabyte file to do that?

[bob]

it really isn't _that_ noisy. I have a bunch of 15K drives in my office machine, and while it is not whisper quiet, it isn't worse than a small refrigerator I also have in my office...

[jwes]

[quote="bob"][quote="M ANSARI"]RAID 0 array will not perform as fast as an solid state drivequote]

That is in theory. In reality raid0 murders solid state drives, because they are all using things like FLASH memory which is ridiculously slow. Yes, if you had a large DRAM disk (Cray Research builds them but they are not cheap) then they would be fast. But if you re-read the original post, he mentioned things like "thumbdrives" and the like and they are horribly slow compared to an array of SCSI drives.

[quote]
Have you tested this for EGTBs? Flash drives have faster access times and slower read times. For small enough blocks, the access times should outweigh the read times.

[smirobth]

it really isn't _that_ noisy. I have a bunch of 15K drives in my office machine, and while it is not whisper quiet, it isn't worse than a small refrigerator I also have in my office...

I had just a single 15000 RPM drive in my machine for a while and it sounded like it was getting ready for take-off; but I installed it myself and perhaps it was resonating with the case. An off the shelf commercial solution would likely not be so bad in terms of noise as my experience.