A fast PRNG for generating hash codes

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: The Mac OS/X man page for random/urandom

Post by bob »

rbarreira wrote:If you think that end users generating crypto keys don't need cryptographically-safe random numbers (which is a contradiction, but ok...) surely you also agree that these random numbers are overkill for chess.
The simple question I answered was "how can one obtain a good seed for a PRNG?" There were suggestions about system time and such. I simply responded that /dev/random (or urandom if you prefer) can be used as a simple way to get a number that will not be very likely to have the same bits if you do it more than once inside a second. That's all there was to it.


My whole point can be stated very simply. Using /dev/random for chess or most other applications cannot be better than using urandom because it will either return numbers with equal quality to urandom (same source), or block. To realize this you only have to be able to read the relevant man page.

It's not about probabilities, it's about simple logic and avoiding unneeded problems without gaining any advantage.
Use the one you prefer. For chess, it doesn't really matter. Either is easier than playing games with low-resolution system timers and such.
User avatar
michiguel
Posts: 6401
Joined: Thu Mar 09, 2006 8:30 pm
Location: Chicago, Illinois, USA

Re: The Mac OS/X man page for random/urandom

Post by michiguel »

bob wrote:
rbarreira wrote:Bob, again you are confusing two different things. The Mac OS /dev/random does NOT block, it does not require true entropy so it's equivalent to /dev/urandom. Therefore it is not good enough for crytographic purposes, which YES, can be important on a Mac despite your earlier statement that no one "in their right mind" would do anything crypto-related there.

It's pointless for me to keep replying when you didn't even read the documentation posted earlier by Steven, and don't make any effort to read my points. Instead you keep replying to things no one said.
OK, my take on this. In linux, as is well-known, /dev/random gives better numbers, but can lag and block if you ask for too many/ /dev/urandom does not block (although it is still slow) but gives a weaker stream of random numbers.

In MacOS, both are the same.

Again, who cares? Nobody is going to need crypto-standard random numbers on a Mac. Regardless of what you are "guessing". You don't need that level of random numbers to produce public/private key pairs and such. Mac's are _not_ going to be used by the true crypto crowd to do any heavy work with really long and independent random numbers.

:shock:
Are you saying that people do not do research doing simulations on Macs?

Miguel

In chess, the chances of /dev/random blocking on linux are essentially zero. Nobody is continually draining it unless you take a really large box (say a 64 processor alpha box or something similar) that might actually be used by the crypto guys. But in chess, not going to happen, if all you want is to seed a PRGN with a different number each time you play a game. I will say it again, in chess, /dev/random is _not_ going top block. Because nobody is going to play chess and be generating tons of random numbers via /dev/random. If you do that, good. That is what, one machine out of maybe 500 million on the planet?

I have used /dev/random many times to get a seed over the years, or to get an n bit number so that I can burn the first N random values from my PRNG so that I start at a different place for book randomness. Never had it block, or cause me any delay of any kind. Because as I said, nobody is using these machines for that kind of work to any measurable extent.

Your worry is "way out there" on the far edge of any significant probability...

Most don't seed things this way at all. But it will work just fine...
wgarvin
Posts: 838
Joined: Thu Jul 05, 2007 5:03 pm
Location: British Columbia, Canada

Re: The Mac OS/X man page for random/urandom

Post by wgarvin »

michiguel wrote::shock:
Are you saying that people do not do research doing simulations on Macs?
I thought he was saying that simulations don't usually need a cryptographically strong source of random numbers. Not many things really need that -- key generation for cryptography, shuffling decks for poker or other games of chance... I can't think of much else, certainly nothing chess-related.
User avatar
sje
Posts: 4675
Joined: Mon Mar 13, 2006 7:43 pm

An excerpt from the Ubuntu 10.10 /dev/random man page

Post by sje »

Code: Select all

RANDOM(4)                               Linux Programmer's Manual                              RANDOM(4)

NAME
       random, urandom - kernel random number source devices

DESCRIPTION
       The  character special files /dev/random and /dev/urandom (present since Linux 1.3.30) provide an
       interface to the kernel's random number generator.  File /dev/random has major  device  number  1
       and  minor  device number 8.  File /dev/urandom has major device number 1 and minor device number
       9.

       The random number generator gathers environmental noise from device  drivers  and  other  sources
       into an entropy pool.  The generator also keeps an estimate of the number of bits of noise in the
       entropy pool.  From this entropy pool random numbers are created.

       When read, the /dev/random device will only return random bytes within the  estimated  number  of
       bits  of  noise in the entropy pool.  /dev/random should be suitable for uses that need very high
       quality randomness such as one-time pad or key generation.  When the entropy pool is empty, reads
       from /dev/random will block until additional environmental noise is gathered.

       A  read  from  the  /dev/urandom device will not block waiting for more entropy.  As a result, if
       there is not sufficient entropy in the entropy pool, the returned values are  theoretically  vul‐
       nerable  to  a cryptographic attack on the algorithms used by the driver.  Knowledge of how to do
       this is not available in the current unclassified literature, but it  is  theoretically  possible
       that  such  an  attack  may  exist.   If  this  is a concern in your application, use /dev/random
       instead.

   Usage
       If you are unsure about whether you should use /dev/random or  /dev/urandom,  then  probably  you
       want  to  use  the  latter.  As a general rule, /dev/urandom should be used for everything except
       long-lived GPG/SSL/SSH keys.

       If a seed file is saved across reboots as recommended above (all major Linux  distributions  have
       done  this since 2000 at least), the output is cryptographically secure against attackers without
       local root access as soon as it is reloaded in the boot sequence, and perfectly adequate for net‐
       work encryption session keys.  Since reads from /dev/random may block, users will usually want to
       open it in nonblocking mode (or perform a read with timeout), and provide some sort of user noti‐
       fication if the desired entropy is not immediately available.

       The  kernel  random-number  generator  is designed to produce a small amount of high-quality seed
       material to seed a cryptographic pseudo-random number generator  (CPRNG).   It  is  designed  for
       security,  not  speed,  and  is  poorly suited to generating large amounts of random data.  Users
       should be very economical in the amount of seed material that they read  from  /dev/urandom  (and
       /dev/random);  unnecessarily  reading large quantities of data from this device will have a nega‐
       tive impact on other users of the device.

       The amount of seed material required to generate a cryptographic key  equals  the  effective  key
       size  of the key.  For example, a 3072-bit RSA or Diffie-Hellman private key has an effective key
       size of 128 bits (it requires about 2^128 operations to break) so a key generator only needs  128
       bits (16 bytes) of seed material from /dev/random.

       While  some safety margin above that minimum is reasonable, as a guard against flaws in the CPRNG
       algorithm, no cryptographic primitive available today can hope to promise more than 256  bits  of
       security,  so  if any program reads more than 256 bits (32 bytes) from the kernel random pool per
       invocation, or per reasonable reseed interval (not less than one minute), that should be taken as
       a sign that its cryptography is not skilfully implemented.

rbarreira
Posts: 900
Joined: Tue Apr 27, 2010 3:48 pm

Re: An excerpt from the Ubuntu 10.10 /dev/random man page

Post by rbarreira »

Thanks Steven.
If you are unsure about whether you should use /dev/random or /dev/urandom, then probably you
want to use the latter. As a general rule, /dev/urandom should be used for everything except
long-lived GPG/SSL/SSH keys.
Yeah, pretty much...
User avatar
sje
Posts: 4675
Joined: Mon Mar 13, 2006 7:43 pm

Re: A fast PRNG for generating hash codes

Post by sje »

I've adjusted the CIL PRNG slightly to ensure that the elements of a state vector are always even and I've provided a routine to initialize a state vector with the Common Lisp random function.

The toolkit uses two PRNG state vectors. The first is cleared to all zero and then used to initialize all the Zobrist hash code vectors; its output is the same each time and on each target platform. The second is initialized with the Lisp random routine and it is used to provide variety in play.

The Common Lisp random routine can be initialized in three ways:

1) It can be initialized with the special value "t" and will crank out a different sequence each time a program is run.

2) It can be initialized with a prior state of itself, and this state can be read from or written to a file.

3) It can be initialized by default from the Lisp runtime and will give the same sequence each time a program is run.

The slightly tricky part is that there is no standard random state structure; each Lisp has its own random state format and internal semantics. And here is the big reason that a Common Lisp chess program needs its own PRNG; externalized hash codes made from a PRNG must be the same regardless of platform to maintain platform independent opening libraries that employ hash code indexing.

The sbcl (Steel Bank Common Lisp) processor has a random state of 627 32 bit integers. That's close to 20,000 bits of state data compared with about 192 bits of state data in the CIL PRNG and only 32 (or fewer) bits in some common PRNGs. I don't know the details of the sbcl PRNG, but I suspect that it is very, very good.
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: The Mac OS/X man page for random/urandom

Post by bob »

michiguel wrote:
bob wrote:
rbarreira wrote:Bob, again you are confusing two different things. The Mac OS /dev/random does NOT block, it does not require true entropy so it's equivalent to /dev/urandom. Therefore it is not good enough for crytographic purposes, which YES, can be important on a Mac despite your earlier statement that no one "in their right mind" would do anything crypto-related there.

It's pointless for me to keep replying when you didn't even read the documentation posted earlier by Steven, and don't make any effort to read my points. Instead you keep replying to things no one said.
OK, my take on this. In linux, as is well-known, /dev/random gives better numbers, but can lag and block if you ask for too many/ /dev/urandom does not block (although it is still slow) but gives a weaker stream of random numbers.

In MacOS, both are the same.

Again, who cares? Nobody is going to need crypto-standard random numbers on a Mac. Regardless of what you are "guessing". You don't need that level of random numbers to produce public/private key pairs and such. Mac's are _not_ going to be used by the true crypto crowd to do any heavy work with really long and independent random numbers.

:shock:
Are you saying that people do not do research doing simulations on Macs?
Nope. I am saying people do not do _cryptographic_ work on a Mac, requiring that level of randomness...



Miguel

In chess, the chances of /dev/random blocking on linux are essentially zero. Nobody is continually draining it unless you take a really large box (say a 64 processor alpha box or something similar) that might actually be used by the crypto guys. But in chess, not going to happen, if all you want is to seed a PRGN with a different number each time you play a game. I will say it again, in chess, /dev/random is _not_ going top block. Because nobody is going to play chess and be generating tons of random numbers via /dev/random. If you do that, good. That is what, one machine out of maybe 500 million on the planet?

I have used /dev/random many times to get a seed over the years, or to get an n bit number so that I can burn the first N random values from my PRNG so that I start at a different place for book randomness. Never had it block, or cause me any delay of any kind. Because as I said, nobody is using these machines for that kind of work to any measurable extent.

Your worry is "way out there" on the far edge of any significant probability...

Most don't seed things this way at all. But it will work just fine...
rbarreira
Posts: 900
Joined: Tue Apr 27, 2010 3:48 pm

Re: The Mac OS/X man page for random/urandom

Post by rbarreira »

bob wrote:Nope. I am saying people do not do _cryptographic_ work on a Mac, requiring that level of randomness...
Of course as stubborn as you are, you're still saying that despite people's explanations and the posted man pages contradicting it.

Note to self: never use a cryptography program written by Robert Hyatt :D
bob
Posts: 20943
Joined: Mon Feb 27, 2006 7:30 pm
Location: Birmingham, AL

Re: The Mac OS/X man page for random/urandom

Post by bob »

rbarreira wrote:
bob wrote:Nope. I am saying people do not do _cryptographic_ work on a Mac, requiring that level of randomness...
Of course as stubborn as you are, you're still saying that despite people's explanations and the posted man pages contradicting it.

Note to self: never use a cryptography program written by Robert Hyatt :D
Just grow up. The man pages have _nothing_ to do with this discussion. Again, for the last time, there are no "crypto" users on a mac...

get over it...
User avatar
sje
Posts: 4675
Joined: Mon Mar 13, 2006 7:43 pm

Re: The Mac OS/X man page for random/urandom

Post by sje »

Well, I'm sure that there are serious crypto users on Macs, but they don' t use /dev/random; they use specialized hardware generators.

http://en.wikipedia.org/wiki/Hardware_r ... _generator

http://en.wikipedia.org/wiki/Comparison ... generators