math/rand: Deterministic random by default is dangerous

[shazow]

Go's standard library is generally considered to have great security-focused defaults (like net/http validating certificates out of the box). In many cases, the default behaviour goes out of the way to prevent the programmer from employing bad practices (such as relying on map key ordering). I believe that the default behavior of math/rand can be improved to better represent these ideals.

There have already been incidents where people didn't realize that math/rand was deterministic by default (example), and even in security-related applications (example). Additionally, tutorials tend to forego mentioning this potentially-catastrophic default (example).

I want to acknowledge that the stdlib documentation mentions this behavior in the second paragraph.

Random numbers are generated by a Source. Top-level functions, such as Float64 and Int, use a default shared Source that produces a deterministic sequence of values each time a program is run. Use the Seed function to initialize the default Source if different behavior is required for each run.

Unfortunately, merely documenting dangerous default behaviour has not proven to be sufficient. Also worth pointing out that the math/rand docs don't mention crypto/rand (this should be fixed too).

To resolve this, I propose that the top-level math/rand functions be seeded by crypto/rand's Reader by default.

Context: PHP's mt_rand was recently torn down for generating only odd numbers when the max value given was too big (a reasonably easy mistake to make; HN thread). Some Twitter discussions started by @richo pointed out that default-deterministic random is a similarly easy mistake to make.

[ALTree]

To resolve this, I propose that the top-level math/rand functions be seeded by crypto/rand's Reader by default.

Would a change in the "random" sequence generated by math/rand functions be a break to the compatibility contract?

See discussion on #8013 and #8731. The community should probably decide about this once and for all.

[cespare]

If math/rand is being used for security-related code then that's already very dangerous, regardless of how the prng is seeded.

[richo]

So to clear a few things up, I was actually frustrated before the mt_rand debacle took hold, although it was illustrative of some of the problems here.

The underlying issue is multi part in my opinion, although some of it is easier solved. In my (humble) opinion having an RNG that isn't a CSPRNG is dangerous in and of itself. Much of Go's philosophy hinges on the fact that people cut corners and don't read docs, and so I tend to think that having a rand that isn't safe for crypto is unsavoury.

On the specific topic of math/rand, I think inheriting rand(3)s API is extremely regrettable. While there are cases for deterministic RNG's, they're almost certainly in the minority (eg, deterministic tests?) and supporting them first seems like a mistake. Especially when the rationale appears to be that "Noone would ever manage to ship software without noticing that the numbers are always the same", the threads that Andrey linked above demonstrate that isn't the case.

Finally, and getting a little offtopic, but while we're talking about Go's RNG's and user safety, having crypto/rand accept a plain old io.Reader seems like a footgun to me. It means that things like http://play.golang.org/p/4NDDz8FSDq work. Worse, it means that http://play.golang.org/p/0e9_Oke1xF works and kills the CSPRNG for anyone consuming your code.

That's the "why", onto the "how". I would propose one of two things:

• rename math/rand to math/deterministic_rand. This has the advantage of forcing a user to comprehend what they're getting themselves into. I rather doubt that the core team is willing to rename a module at this late stage of the game so;
• Seed math/rand with crypto/rands reader, as proposed by @shazow above.

Cheers

richo

[bnagy]

Much of Go's philosophy hinges on the fact that people cut corners and don't read docs

Settle down. If anything, the Go "philosophy" is the opposite.

I think that seeding math/rand from crypto/rand will make the problem even worse, because then one could make an (ill-advised) argument for its security.

I feel like providing NO options for a non-CSPRNG in a stdlib is a little opinionated, even for Go.

I doubt it would be palatable, but I would quite like to see an insecure package, in the same vein as subtle - in other words, you need a very good excuse to use it. We could hide all kinds of skeletons in this closet - RC4, MD5, DES ... and the PRNG. It would be difficult for anyone to mistakenly use insecure/rand for, eg, a password generator.

[cespare]

having an RNG that isn't a CSPRNG is dangerous in and of itself. Much of Go's philosophy hinges on the fact that people cut corners and don't read docs, and so I tend to think that having a rand that isn't safe for crypto is unsavoury.

I doubt people that don't read the docs enough to notice the difference between math/rand and crypto/rand would otherwise then be able to use crypto/rand to implement crypto primitives securely.

Can you point at other popular language standard libraries whose only PRNG is a CSPRNG? There are plenty of use cases for fast, non-crypto PRNGs. (The math/rand PRNG isn't particularly fast, but that's a separate issue.)

rename math/rand to math/deterministic_rand

Will not happen due to Go's compatibility promise

Seed math/rand with crypto/rands reader

That might lead people to think that math/rand has security properties that it doesn't.

[richo]

Re Go's philosophy- we can have it out on IRC if you like. Tl;dr I think this philosophy is actively positive because it turns out people are fallible and computers are better at diligently checking things.

I would be onboard with an insecure, although I suspect that the chances of upstreaming it are pretty slim for the compatibility reasons above.

I mostly don't buy the argument that people can't implement safe crypto primitives. It seems like conflating the concerns here. A password generator a perfect of an example that can be implemented safely by nearly anyone, but impossible to be implemented safely on top of a deterministically seeded RNG.

I also get the impression from nearly everything about Go's stdlib that it's optimised for the 90% case, with the suggestion that if you're in the 10% outliers you might need to dig a bit more to do what you want (Or reconsider whether what you're trying to do is actually correct). That seems to me like given the state of entropy pools on modern systems, backing onto a CSPRG should be safe, and in the event that you do manage to exhaust the pool, or that your program because untenably slow you can seek out alternative solutions.

It seems to me that cases where you can't use system RNG, or have specific needs about it's entropy would always fall into the 10% case.

[ianlancetaylor]

The question of whether math/rand should keep the same sequence of values it has had in the past is a question to raise on the golang-dev mailing list.

The Go 1 compatibility promise means that we can't rename or significantly change the math/rand package. We could add more documentation to it.

[rogpeppe]

FWIW I am +1 on randomly seeding math/rand at init time (probably just from the current time). Because it's global, unless you're writing a trivial program, repeatability isn't something you should be relying on anyway because any other package is free to get a random number at any time, disrupting your sequence. If you need a deterministic RNG, you can always make your own(which is faster too).

[cespare]

@rogpeppe that sounds like an improvement to me too, but can the math/rand API be changed like that at this point? Consider this language in the docs:

Top-level functions, such as Float64 and Int, use a default shared Source that produces a deterministic sequence of values each time a program is run. Use the Seed function to initialize the default Source if different behavior is required for each run.

Given the discussion in #8013, the emerging consensus seemed to be that it would be okay to change the deterministic sequence to a different deterministic sequence if necessary (it ended up not being necessary), but any more than that, such as random seeding, seems well outside the compatibility promise.

[shazow]

Another reference is #11372, for randomizing the scheduler to prevent people from depending on the current order that goroutines happen to run by default. This is in anticipation of future scheduler changes affecting the natural order of goroutines.

In general, it seems like a good idea to randomize things by default to prevent people from becoming dependent on implementation-specific values, whether intentionally or unintentionally.

[rogpeppe]

@cespare You are probably right. Much as I think the current behaviour is not that useful, there probably are programs out there that rely on the deterministic behaviour, generating random numbers from only a single known thread of control. That's a pity because that's exactly the case where a newly made Source is appropriate.

[colinnewell]

While I agree with the principle of making sure that people use a secure random when it's appropriate, secure random isn't the only random that is necessary. Simply replacing math/random with a secure random is likely to break existing promises about number distributions.

Adding the reference to the crypto/rand in the documentation would definitely be a good idea.

If you make common tasks like random password/token/key generation really easy via the crypto/rand library then you are likely to encourage a lot more people down the secure path. This may well be in a separate library, but so long as it's really really easy to use and google, it should help guide people onto the correct path.

[shazow]

Seems there is agreement that, at minimum, we need to improve the rand/math docs to urge the consideration of the crypto/rand package for security-sensitive work. I opened a CL here: https://go-review.googlesource.com/#/c/12900/, phrasing suggestions are welcome.

[gopherbot]

CL https://golang.org/cl/12900 mentions this issue.