proposal: spec: change all int types to panic on wraparound, overflow

[bcmills]

I know this has been discussed before, but I didn't see a specific proposal filed for it yet and I think it's important.

Unexpected integer overflow can lead to serious bugs, including bugs in Go itself. Go's bounds-checking on slices and arrays mitigates some of the harmful effects of overflow, but not all of them. For example, programs that make system calls may pass data structures into the kernel, bypassing Go's usual bounds checks. Programs that marshal data-structures to be sent over the wire (such as protocol buffers) may send silently-corrupted data instead of returning errors as they ought to. And programs that use unsafe to access addresses with offsets are vulnerable to exactly the same overflow bugs as in C.

In my experience, Go programs and libraries are often written assuming "reasonable inputs" and no overflow. For such programs, it would be clearer for overflow to cause a run-time panic (similar to dividing by zero) rather than silently wrapping around. Even in the case where the unintended overflow is subsequently caught by a slice bounds check, reporting the error at the overflowing operation rather than the slice access would make the source of the bug easier to diagnose.

The potential performance impact of this proposal is similar to bounds-checking in general, and likely lower than using arbitrary-precision ints (#19623). The checks can be omitted when the compiler can prove the result is within bounds, any new branches will be trivially predictable (they'll occupy some CPU resources in the branch-predictor but otherwise add little overhead), and in some cases the checks might be able to use bounds-check instructions or other hardware traps.

For the subset of programs and libraries that intentionally make use of wraparound, we could provide one of several alternatives:

1. "comma, ok" forms or "comma, carry" forms (proposal: spec: extend comma-ok expressions to + - * / arithmetic #6815) that ignore overflow panics, analogous to how the "comma, ok" form of a type-assertion ignores the panic from a mismatched type.
2. Separate "integer mod 2ⁿ" types (requiring explicit conversions from ordinary integer types), perhaps named along the lines of int32wrap or int32mod.
3. Implicit wrapping only for unsigned types (uint32 and friends), since they're used for bit-manipulation code more often than the signed equivalents.

Those alternatives could also be used to optimize out the overflow checks in inner-loop code when the programmer has already validated the inputs by some other means.

---

[Edit: added this section in response to comments.]

Concretely, the proposed changes to the spec are:

Integer operators

For two integer values x and y, the integer quotient q = x / y and remainder r = x % y satisfy the following relationships:

[…]

As an exception to this rule, if the dividend x is the most negative value for the int type of x, the quotient q = x / -1 is equal to x (and r = 0).

[…]

The shift operators shift the left operand by the shift count specified by the right operand. They implement arithmetic shifts if the left operand is a signed integer and logical shifts if it is an unsigned integer. The result of a logical shift is truncated to the bit width of the type: a logical shift never results in overflow. Shifts behave as if the left operand is shifted n times by 1 for a shift count of n. As a result, x << 1 is the same as x*2 and x >> 1 is the same as x/2 but truncated towards negative infinity.

[…]

Integer overflow

If the result of any arithmetic operator or conversion to an integer type cannot be represented in the type, a run-time panic occurs.

An expression consisting of arithmetic operators and / or conversions between integer types used in an assignment or initialization of the special form

v, ok = expr
v, ok := expr
var v, ok = expr
var v, ok T1 = expr

yields an additional untyped boolean value. The value of ok is true if the results of all arithmetic operators and conversions could be represented in their respective types. Otherwise it is false and the value of v is computed as follows. No run-time panic occurs in this case.

For unsigned integer values, the operations +, -, *, and << are computed modulo 2ⁿ upon overflow, where n is the bit width of the unsigned integer's type. Loosely speaking, these unsigned integer operations discard high bits upon overflow, and programs may rely on ``wrap around''.

For signed integers, the operations +, -, *, and << are computed using two's complement arithmetic and truncated to the bit width of the signed integer's type upon overflow. No exception is raised as a result of overflow. A compiler may not optimize code under the assumption that overflow does not occur. For instance, it may not assume that x < x + 1 is always true.

If the dividend x of a quotient or remainder operation is the most negative value for the int type of x, evaluation of x / -1 overflows and its result upon overflow is equal to x. In contrast, evaluation of x % -1 does not overflow and yields a result of 0.

[…]

Conversions between numeric types

For the conversion of non-constant numeric values, the following rules apply:

1. When converting between integer types, if the value is a signed integer, it is sign extended to implicit infinite precision; otherwise it is zero extended. If the value cannot be represented in the destination type, an overflow occurs; see the section on integer overflow. Upon overflow, the result is truncated to fit in the result type's size. For example, if v := uint16(0x10F0), then w, _ := uint32(int8(v)) results in w == 0xFFFFFFF0.

---

This proposal is obviously not compatible with Go 1, but I think we should seriously consider it for Go 2.

[dr2chase]

This simplifies bounds check elimination, since guarding-against/proving-impossibility-of overflow in indexing calculations can be tricky.

[randall77]

Just a datapoint, the code in https://golang.org/src/runtime/hash64.go takes advantage of overflow in almost every line of source (for the uintptr type).

Maybe we can't do this until Go 2, but we can do experiments to get some data about it now. We could hack panic on overflow into the compiler today. What happens performance-wise with the go1 benchmarks? How many false positives are there? How many true positives are there?

[griesemer]

From a programmer's point of view, proposal #19623 is a significant simplification (wrap around and overflow disappear, and the new int type is both simpler in semantics and more powerful in use), while this proposal is a significant complication (wrap around and overflow now panic, and the new int type is both more complex in semantics and more difficult to use).

One of Go's ideas is to reduce the intrinsic complexity of the programming language at hand while at the same time provide mechanisms that are general and powerful and thus enable programmer productivity and increased readability. We need to look at language changes not from a compiler writer's point of view, but from a programmer's productivity point of view. I think this proposal would be a step backward in the philosophy of Go.

I am also not convinced about the claim that overflow checking is "likely much lower than using arbitrary-precision ints": The cost of arbitrary precision ints is there when one actually uses them, otherwise their cost is similar to what needs to be done for bounds/overflow checking (it's the same test, essentially). There's a grey area for ints that use all 64 (or 32) bits as they will become "big ints" internally (at least one bit is usually reserved to implement "tagged ints" efficiently) - but in code that straddles this boundary and if it matters one might be better off using a sized intxx type anyway. Finally, there's a GC cost since the garbage collector will need to do extra work. But all that said, dealing with overflow panic will also require extra code, and that has to be written by each programmer by hand. It is also much harder to verify/read that code.

I'm not in favor of this proposal.

[dr2chase]

I think either proposal is an improvement over the status quo. Programmers who assert the nonexistence of overflow will write the same code they do today, so no cognitive overhead there, and I won't have to worry about silent errors if they're wrong.

[bcmills]

@griesemer

proposal #19623 is a significant simplification […], while this proposal is a significant complication

I believe that the two proposals are compatible (and even complementary). We could make int and uint arbitrary-precision types (to make default behavior simpler), and also make the sized integer types panic on overflow (to make complex behavior safer).

But all that said, dealing with overflow panic will also require extra code, and that has to be written by each programmer by hand. It is also much harder to verify/read that code.

Could you elaborate on this point? I would expect that most code would either not handle the panic, or use a wrapping integer type explicitly. Even the latter option does not seem like a lot of extra code.

[bcmills]

@randall77

Just a datapoint, the code in https://golang.org/src/runtime/hash64.go takes advantage of overflow in almost every line of source (for the uintptr type).

That is a nice data point, and I think it nicely illustrates the three options I propose for intentional wraparound. Consider this snippet:

	h := uint64(seed + s*hashkey[0])
tail:
	switch {
	case s == 0:
	case s < 4:
		h ^= uint64(*(*byte)(p))
		h ^= uint64(*(*byte)(add(p, s>>1))) << 8
		h ^= uint64(*(*byte)(add(p, s-1))) << 16
		h = rotl_31(h*m1) * m2
	case s <= 8:
		h ^= uint64(readUnaligned32(p))
		h ^= uint64(readUnaligned32(add(p, s-4))) << 32
		h = rotl_31(h*m1) * m2

1. With the "comma, ok" option it becomes unwieldy: there are many lines which combine the ^ with shifting or multiplication, and it is the latter which may overflow (so using the "comma, ok" form requires splitting lines):
   Edit: See proposal: spec: change all int types to panic on wraparound, overflow #19624 (comment) below. If we apply the _, ok to the entire expression tree rather than just the immediate expression, it's not unwieldy at all.

	h := uint64(seed + s*hashkey[0])
tail:
	switch {
	case s == 0:
	case s < 4:
		h ^= uint64(*(*byte)(p))
		x, _ := uint64(*(*byte)(add(p, s>>1))) << 8
		h ^= x
		x, _ = uint64(*(*byte)(add(p, s-1))) << 16
		h ^= x
		x, _ = h * m1
		h, _ = rotl_31(x) * m2
	case s <= 8:
		h ^= uint64(readUnaligned32(p))
		x, _ := uint64(readUnaligned32(add(p, s-4))) << 32
		h ^= x
		x, _ = h * m1
		h = rotl_31(x) * m2

2. With explicitly-wrapping types, only the conversions (which are mostly already present in the code) need to change:

	h := uint64mod(uintptrmod(seed) + uintptrmod(s)*hashkey[0])
tail:
	switch {
	case s == 0:
	case s < 4:
		h ^= uint64mod(*(*byte)(p))
		h ^= uint64mod(*(*byte)(add(p, s>>1))) << 8
		h ^= uint64mod(*(*byte)(add(p, s-1))) << 16
		h = rotl_31(h*m1) * m2
	case s <= 8:
		h ^= uint64mod(readUnaligned32(p))
		h ^= uint64mod(readUnaligned32(add(p, s-4))) << 32
		h = rotl_31(h*m1) * m2

3. With implicit wrapping only for unsigned types, that function wouldn't change at all, although I think that also demonstrates that the safety advantages of detecting overflow diminish with that approach (since uintptr is a fairly common type to see in code using unsafe).

[griesemer]

@bcmills The point of the sized integer types is a) to be able to control actual space consumed when laid out in memory, and b) often enough that they do wrap around. There's tons of code that makes use of that. Almost any left-shift operation would become more complicated if there wasn't silent overflow. Thus, a lot of code would have to deal with overflow.

[bcmills]

The point of the sized integer types is a) to be able to control actual space consumed when laid out in memory, and b) often enough that they do wrap around.

That's part of my point? At the moment, the sized integer types conflate together (a) and (b). I'm proposing that we make them orthogonal, not eliminate (b).

Almost any left-shift operation would become more complicated if there wasn't silent overflow.

Perhaps that's a good argument for making the left-shift operator not panic? (None of the other bitwise operators can overflow, and that would make the shift operator somewhat less redundant with multiplication.)

[griesemer]

@bcmills Perhaps. My point is that this all adds extra complexity to the language where I am not convinced that we need more. We need less. Most people couldn't care less about overflow and simply want integers that "just work".

[bcmills]

Most people couldn't care less about overflow and simply want integers that "just work".

That's also part of my point? The fact that most people couldn't care less about overflow is what leads to the bugs in the first place. Most people couldn't care less about bounds-checking either, but Go has bounds checks nonetheless.

I agree that 'integers that "just work"' is a desirable goal, and ideally I would like to see this proposal combined with an arbitrary-precision int type. However, it's not obvious to me that that will be sufficient to make a dent in the incidence of overflow bugs in practice.