Description
The original math/rand package had a number of flaws. Some of the flaws in the API have been addressed, such as the addition of the Source64 interface, but the most important issue remains: the default Source is poor.
The Source has several problems.
- It has no clear provenance (it came from Plan 9, but its history before that is unknown).
- It is a linear feedback shift register, which is known to have flaws uncovered by various empirical tests.
- It is difficult and expensive to seed.
- It is enormous! The shift register itself comprises 607 64-bit words, a total of 4856 bytes.
This last point is not fatal if there is only one instance, but it couples poorly with another fundamental and perhaps unfixable problem: bad behavior when accessed concurrently. It must be protected with a mutex, but if it were much smaller (and cheaper to seed) it would be practical instead to have each client in the address space access a separate instance, seeded uniquely.
To put it another way, with a separate Source for each goroutine, no locking would be required, but the current implementation is impractically large for such an approach.
The proposal here is in several parts.
First, we create an alternate Source implementation. Our choice is the Permuted Congruential Generator (PCG) designed and thoroughly tested by O'Neill in:
PCG: A Family of Simple Fast Space-Efficient Statistically Good
Algorithms for Random Number Generation
Melissa E. O’Neill, Harvey Mudd College
http://www.pcg-random.org/pdf/toms-oneill-pcg-family-v1.02.pdf
A 128-bit PCG generator producing a 64-bit output is only two words of memory but provably has a period of 2**128, is cheap to seed, and efficient. (A 64-bit generator would be noticeably cheaper, but a period of only 2**64 is deemed too small for modern hardware). It is practical and reasonable for every goroutine to have a local Source (a simple seeding algorithm is to use the address of the Source as the seed).
An implementation of this generator, specifically PCG XSL RR 128/64 (LCG), is checked in to golang.org/x/exp/rand as its default Source.
Second, we work on the compiler, possibly through math/bits, to make it a little more efficient. The current implementation, in pure Go, is somewhat slower than math/rand, but could be comparably fast or even faster given compiler support for the 128-bit multiply inside.
The third part is the critical piece.
We propose to make this generator the default one in math/rand for Go 2. The Go 1 compatibility decree could allow it to happen today, but we have chosen in the past to take the guarantee more literally than is expressed by also requiring that the generated stream of bits will never change. This decision was made when a fix was done, but enough tests in production were discovered that depended on the exact stream that it was deemed simpler to lock down the implementation rather than to fix all the tests.
However, with Go 2 on the horizon, there is enough time and understanding and process to offer an opportunity to use rolling out an improved, API-compatible version of math/rand as an early test case for Go 2. Changing math/rand to use PCG is strictly incompatible, but not in a way that the Go 1 decree actually prohibits. A soft breakage at this level would be an excellent step on the way to understanding what it takes to roll out a breaking change for Go 2.
In essence, the proposal is to use the fixing of math/rand as a test case for introducing breaking changes in Go 2. And a much better random number generator would result, too: the breaking change is worth making.
The precise sequence of steps to roll it out needs discussion, but one possible outline is like this:
- Add PCG as a secondary source to
math/rand, but not as the default. - Make it available under an alternate name;
PCGSourceis the obvious. - Provide an alias for the current source, say
LFSRSource. - Optional: Provide a gofix that updates existing code to use
LFSRSourcerather thanSource. SinceSourceis usually hidden, accessed throughRand, this may be ineffectual unless we also provide PCG functions explicitly forInt32, etc. Or perhaps provide LFSR versions and rotate towards those in legacy code. - Make announcements of what's to come.
- At some point, flip the alias so
Sourcepoints toPCGSourceandLFSRSourceis no longer the default. - In parallel with this work, make
math/bitssupport its operations more efficiently through the compiler. - Just prior to Go 2, delete
LFSRSource.
At some point along the way, Source64 should become the default, under a similar sequence of events. Minor cleanups could occur too, tidying up the API and fixing some old-style names like Intn not IntN.
There is much detail that remains to be worked out; this proposal is only an outline.