math/rand: rand.Float64 is not purely uniform

[fabienlefloch]

The current implementation (as of Go 1.5) of rand.Float64 is:

func (r *Rand) Float64() float64 {
  f := float64(r.Int63()) / (1 << 63)
  if f == 1 {
    f = 0
  }
  return f
}

As the mantissa of a Float64 is on 52 bits, there are a lot of numbers (2^11 ?) that were initially mapped to 1 that will effectively become 0. The Float64() method therefore does not preserve the uniform property of the underlying Int63() method.

This is related to issues #4965 and #6721 .

A better alternative, even if this implies to trade numbers under machine epsilon against uniformity comes from the prototypical code of Mersenne Twister 64 at http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/VERSIONS/C-LANG/mt19937-64.c :

/* generates a random number on [0,1)-real-interval */
func (r *Rand) Float64() float64 {
    return (float64(r.Int63()) >> 10) * (1.0/9007199254740992.0)
}

/* generates a random number on (0,1)-real-interval */
func (r *Rand) Float64Open() float64 {
    return (float64(r.Int63())  >> 11) + 0.5) * (1.0/4503599627370496.0)
}

An interesting consequence is that this preserves symmetry around 0.5. It's also quite a bit faster on my machine, as the if statement of the original algorithm deteriorate performance significantly (not entirely sure why).

[stemar94]

There is a comment in the source code:

    // A clearer, simpler implementation would be:
    //  return float64(r.Int63n(1<<53)) / (1<<53)
    // However, Go 1 shipped with
    //  return float64(r.Int63()) / (1 << 63)
    // and we want to preserve that value stream.
    //
    // There is one bug in the value stream: r.Int63() may be so close
    // to 1<<63 that the division rounds up to 1.0, and we've guaranteed
    // that the result is always less than 1.0. To fix that, we treat the
    // range as cyclic and map 1 back to 0. This is justified by observing
    // that while some of the values rounded down to 0, nothing was
    // rounding up to 0, so 0 was underrepresented in the results.
    // Mapping 1 back to zero restores some balance.
    // (The balance is not perfect because the implementation
    // returns denormalized numbers for very small r.Int63(),
    // and those steal from what would normally be 0 results.)
    // The remapping only happens 1/2⁵³ of the time, so most clients
    // will not observe it anyway.

[fabienlefloch]

That's an interesting comment, but the consequence of those choices is a non uniform distribution, where 0 and numbers near 0 are over-represented.

[dr2chase]

A quick test of

// Float64 returns, as a float64, a pseudo-random number in [0.0,1.0).
func (r *Rand) Float64() float64 {
    // Change from Go 1 -- make results be uniform.
    return (float64(r.Int63() >> 10)) * (1.0 / 9007199254740992.0)
}

// Float32 returns, as a float32, a pseudo-random number in [0.0,1.0).
func (r *Rand) Float32() float32 {
    // Change from Go 1 -- make results be uniform.
    return (float32(r.Int63() >> 39)) * (1.0 / 16777216.0)
}

shows a non-uniform distribution of the LSB -- it is 25% 1, instead of 50% 1.

[fabienlefloch]

Isn't it wrong to look at the LSB for floating point numbers?
Floating point numbers are not evenly distributed, there are much more of them near 0 than near 1. This is why truncation to 53 bits (the mantissa) is necessary to have a uniform distribution.

[dr2chase]

As I understand it, staring with a 53-bit integer and then dividing it by 2-to-the-53 means that there are floating point numbers that will occur with probability zero. Any "random" value less than 1/2 (252/253) will not have its LSB set, for example. I think this is also a flaw in the original algorithm, but for a smaller set of FP numbers.

I don't actually know what is officially "right" here because I am not an expert, but right now neither of these algorithms is doing much for me. I recall from some earlier work that one way to do this is to take a 64-bit random number, split it into 12 and 52-bit parts, stuff 52-bits into mantissa, and then count leading zeroes on the 12-bit part to derive your exponent. If the 12-bit part is all zero, pull more 64-bit random numbers till you get one that is not zero or till you hit (about, up to fenceposts) 1024 of them in which case you call it zero, and use the number of leading zeroes to derive your exponent. Since you'd only do second RNG call 1 time out of 4096, this could be plenty efficient since it involves no FP ops at all (though we might hope that multiplication and division by powers of two would be very efficient).

I am somewhat creeped out that both algorithms are guaranteed not to hit certain FP values.

[fabienlefloch]

I would not worry about not hitting certain FP values, especially since FPs are not uniformly distributed by nature (much more towards 0 than towards 1). What's important is that the FP values generated have a uniform distribution. If you restrict yourself to the 53 bits, then the FP numbers are uniformly spaced in (0,1), so it's easy to grasp that the FP generator has to be uniform if the underlying integer generator is uniform. Yes you loose the small numbers around 0 (less than machine epsilon) but there is no way to be uniform and keep those.

Maybe it's not a very good argument, but I would trust Matsumoto and the official Mersenne-Twister implementation as he is a very prominent researcher in the area.

[fabienlefloch]

maybe this paper could help understanding the idea of uniform random floating point numbers:
http://www.doornik.com/research/randomdouble.pdf

[fabienlefloch]

Anybody doing Monte-Carlo simulations will want the Matsumoto like behavior. I don't really see any other use case for a Float64 function than Monte-Carlo simulations.
If backward compatibility is an issue, a Float53() function would be nice

[dr2chase]

Any particular reason not to instead use

// Float64 returns, as a float64, a pseudo-random number in [0.0,1.0).
func (r *Rand) Float64() float64 {
    // Change from Go 1 -- make results be uniform.
    return float64(r.Int63()) * 1.0842021724855043e-19
}

Provenance of funny constant: https://play.golang.org/p/FSkOkffu_2

This gets you uniformity, and hits more floating point numbers.
It does however sometimes cause a 1eps deviation from the former value
stream.

[fabienlefloch]

look at the comment from stemar94, 0 is underrepresented under this algo. You just can't hit more random points and have a uniform distribution appropriate for simulation.

[dr2chase]

The value stream argument seems to be winning at this end (i.e., I floated the possibility of an RNG that differed by 1eps in many instances, and that was judged to be not worth the trouble), but I agree that the wrap-back from 1 overrepresents 0, by I think a factor of 1<<9 for Float64 and 1<<38 for Float32 -- I may have a fencepost error in my exponents, the derivation is that the highest two values for Float64 are 1 and 1-1/(1<<53), there are 1024 points between them in the integer random stream, and each point gets half, versus a single point for 1/(1<<63) and 0. Wrapping 1 to 0 maps 512 points from the integer stream to 0, which is a substantial overrepresentation, and egregiously so for Float32.

To (mostly) preserve the value stream, it appears that the least-awful method is to replace the zero substitution with try-another-number. It benchmarked the same, though of course this was just a microbenchmark.

I don't understand the claim that zero is overrepresented. The Mersenne twister RNG will generate 1/(1<<53) and 0 both at p=1/(1<<53), this algorithm generates 1/(1<<63) and 0 both at p=1/(1<<63). In both cases the probability of zero and its nearest neighbor are the same (and p(0) is lower in this algorithm because all of the points between 0 and 1/(1<<54) get hits, where in Mersenne twister they instead round to zero).

[rsc]

At this point I don't believe we should change the value stream generated by (*Rand).Float64 in any significant way. The split of rand.Source (the source of non-determinism) from rand.Rand (the algorithms on top) means that people expect that only rand.Source might differ from release to release. If people use the same rand.Source output as input to rand.Rand, they expect to get the same outputs. rand.Rand should not itself be a source of non-determinism, not even from release to release, if at all possible.

When the implementation of (*Rand).Float64 returned 1, that was obviously a mistake and in contradiction to the docs, so it merited a breaking change.

The implementation of (*Rand).Float64 is not perfect. My apologies. But I believe it's not awful enough to merit a breaking change.

The if f == 1 { return 0 } fix is also not perfect. Again, my apologies. But even here I'm not convinced it's awful enough at this point to merit a breaking change.

We may be able to loop and grab another value, but I'm not even convinced that's necessary at this point.

More generally, the specified API for (*Rand).Float64 is incomplete. All it says is “Float64 returns, as a float64, a pseudo-random number in [0.0,1.0) from the default Source.” It says nothing about uniformity, which is good, because floating point numbers themselves are not uniformly distributed in [0, 1).

In practice, if you care deeply about the properties of your random floating-point numbers, you should probably write your own wrapper around r.Int63() instead of depending on unspecified details of a library implementation.