proposal: spec: improve for-loop ergonomics

[bcmills]

Motivation

Go's for-loops encourage difficult-to-read code.

1. The Go 1 loop syntax sets the wrong defaults. The syntax is optimized for three-part ForClause loops, but range loops are far more common (by a ratio of nearly 4:1 in the code I sampled) and arguably ought to be viewed as the “default”.

   • The three-part ForClause form is nearly always used for iterating one variable over sequential integers. That puts the interesting part — the condition — in the middle, where it is the hardest to find.

     (For the rare other cases, it is always possible to express a three-part ForClause as an equivalent one-part ForClause with an extra scope block. Loops that use continue require care, but continue in a three-part non-integer loop is especially rare.)

   • Nothing else in the language has a three-part form, and the existence of the three-part for loop precludes a more useful two-part alternative (for APIs such as io.Reader), because it would be too easy to confuse a two-part loop with a three-part one.

2. The range keyword is confusing to newcomers.

   • In set theory, range means "image" or "codomain", but the single-value version of a Go 1 range loop instead iterates over the domain of the slice, map, or array. That makes the single-value form confusing, especially when the index and element types are mutually assignable (https://play.golang.org/p/c-lWoTI_Z-Y) or when the value is used as an interface{} (https://play.golang.org/p/cqZPSHZtuwH).

   • In some other programming languages (such as Python), range refers to a sequence of points in a numerical interval, evoking line segment range or statistical range. In contrast, the Go range keyword doesn't have anything to do with numerical intervals, except to the extent that slice indices happen to be intervals.

   • The fact that range modifies the semantics of := and = is surprising. The only other Go operator that modifies the semantics of another operator is = itself, which (beyond the , ok idiom) modifies the semantics of the index operator ([]) for map assignments. (I think we should fix that too; see proposal: spec: disallow NaN keys in maps #20660 (comment).)

     It is rarely useful to have a range loop assign to existing variables, and we could address that use-case more cleanly with a finally or else keyword anyway.

3. Eliminating the range keyword would allow us to fix variable capture (proposal: spec: redefine range loop variables in each iteration #20733) in a way that does not unexpectedly change the semantics of for-loops written in the Go 1 style. (That is, old-style loops would no longer compile, instead of successfully compiling to something different from before.)

---

Proposal

1. Remove the range keyword and the three-part loop form.

2. Make the range form of the for loop more concise, and add a two-part form and optional else block.

   • For the one-part form:

     • If the first part is of the form x : z or x, y : z, it introduces new variables x and y (as applicable), which take the value of each successive element of z. The one-variable form can be used only for channels and numeric intervals (see interval below). The two-variable form can be used only for maps, slices, strings, and arrays.

     • Otherwise, the first part must be a boolean expression and specifies the Condition of the loop.

   • The new two-part form parallels the two-part form of switch. The first part is an arbitrary statement (usually a short variable declaration) to be evaluated before every iteration, and the second part is the Condition:

     for x, err := f(); err == nil {

   • An else block may follow a loop that has with a Condition. Control transfers to the else block when the condition is false (like else in Python loops). The variables declared in the first part of the two-part form remain in scope for the else block.

     • (If we don't like the way else reads, we could drop that part entirely, or use some other keyword — such as finally — and/or tweak the semantics, for example by also transferring control to the block in case of a break.)

3. Add a built-in pseudofunction interval to replace the vast majority of existing 3-part loops.

   • interval(m, n) returns a container that iterates over [m, n) by increments of 1.

   • interval(m, n, step) returns a container that iterates from m (inclusive) to n (exclusive) by step.

---

Examples

Simple conditions

Loops with just a Condition remain the same as in Go 1.

for a < b {
	a *= 2
}

---

for len(h) > 0 {
	x := heap.Pop(h)
	f(x.(someType))
}

Ranges

Range loops lose a little bit of boilerplate, and gain a closer resemblance to for-each loops in other languages with C-like syntax (such as C++ and Java).

for i, s := range a {
	g(i, s)
}

becomes

for i, s : a {
	g(i, s)
}

---

(from https://github.com/golang/go/wiki/SliceTricks#filtering-without-allocating)

b := a[:0]
for _, x := range a {
	if f(x) {
		b = append(b, x)
	}
}

becomes

b := a[:0]
for _, x : a {
	if f(x) {
		b = append(b, x)
	}
}

Intervals

Simple numeric intervals move the limit closer to the end of the line (where it is easier to find), and in some cases drop the need for redundant variables.

for i, n := 0, f(x); i < n; i++ {
	g(i)
}

becomes

for i : interval(0, f(x)) {
	g(i)
}

---

for n := runtime.GOMAXPROCS(0); n > 0; n-- {
	go …
}

becomes

for n : interval(runtime.GOMAXPROCS(0), 0, -1) {
	go …
}

---

(from https://github.com/golang/go/wiki/SliceTricks#reversing, noting that the original goes out of its way
— and loses some clarity in the process — to avoid re-evaluating len(a)/2 at each iteration)

for i := len(a)/2-1; i >= 0; i-- {
	opp := len(a)-1-i
	a[i], a[opp] = a[opp], a[i]
}

becomes

for i : interval(0, len(a)/2) {
	opp := len(a)-1-i
	a[i], a[opp] = a[opp], a[i]	
}

or

for i, _ : a[:len(a)/2] {
	opp := len(a)-1-i
	a[i], a[opp] = a[opp], a[i]	
}

Iterators

Iterator patterns shed boilerplate and/or levels of indentation.

for {
	n, err := r.Read(p)
	if err != nil {
		if err != io.EOF {
			return err
		}
		return nil
	}
	f(p[:n])
}

becomes

for n, err := r.Read(p); err == nil {
	f(p[:n])
} else if err != io.EOF {
	return err
}
return nil

---

iter := Begin()
for x, ok := iter.Next(); ok; x, ok = iter.Next() {
	f(x)
}

becomes

iter := Begin()
for x, ok := iter.Next(); ok {
	f(x)
}

Lists

Loops iterating over certain custom containers (such as linked lists) become a bit more awkward.
(On the other hand, I would argue that they were awkward to begin with — and they could be fixed by a further change to allow types to implement the range-like behavior directly.)

for e := l.Front(); e != nil; e = e.Next() {
	f(e)
}

becomes

e := l.Front()
for e != nil {
	f(e)
	e = e.Next()
}

[bcmills]

Come to think of it, I think this proposal is actually Go 1 compatible (except for actually removing range). The proposed new forms are all invalid syntax today.

[slrz]

I think the time to fundamentally change how for-loops work has passed many years ago.

[davecb]

As a first qualifying question, are there correctness-preserving transforms between the old and new forms, such that if I write any old form the compiler cannot possibly get it wrong, and source rewriting (fix) can always create the new one?
Secondly, can I do exactly the same thing in my head when I inadvertantly think FOR 1=1,10,1, that will give me the proper form (;-))

[bcmills]

are there correctness-preserving transforms between the old and new forms, such that if I write any old form the compiler cannot possibly get it wrong, and source rewriting (fix) can always create the new one?

Yes. The general form of the rewrite is:

for InitStmt; Condition; PostStmt {
	StatementList1
	if Expression {
		continue
	}
	StatementList2
}

to

{
	InitStmt
	for Condition {
		StatementList1
		if Expression {
			PostStmt
			continue
		}
		StatementList2
		PostStmt
	}
}

That general form can often be simplified.

• If InitStmt, Condition, and PostStmt form an interval, interval can be used instead.
• If InitStmt doesn't shadow existing variables, the outer scope block can be removed.
• If PostStmt is long compared to StatementList2, the if Expression can be inverted and the continue dropped:

  StatementList1
  if !Expression {
  	StatementList2
  }
  PostStmt
  

---

can I do exactly the same thing in my head

That I can't answer. :)

[TocarIP]

How will something like this look?

// From crypto/tls/tls_test.go
 for delay := time.Millisecond; delay <= 64*time.Millisecond; delay *= 2 {
                if err = testConnReadNonzeroAndEOF(t, delay); err == nil {
                        return
                }
        }

[bcmills]

@TocarIP I would consider the larger block

go/src/crypto/tls/tls_test.go

Lines 222 to 228 in 010579c

	var err error
	for delay := time.Millisecond; delay <= 64*time.Millisecond; delay *= 2 {
	if err = testConnReadNonzeroAndEOF(t, delay); err == nil {
	return
	}
	}
	t.Error(err)

and write it in the two-part form:

delay := time.Millisecond
for err := testConnReadNonzeroAndEOF(t, delay); err != nil && delay < 64 * time.Millisecond {
	delay *= 2
} else if err != nil {
	t.Error(err)
}

That has a number of benefits:

• The “expected” exit condition for the loop moves from the loop body to the actual Condition position of the for statement.
  • That eliminates the need for a return from the loop body, which makes it compose better with other code.
  • See also CodeReviewComments#indent-error-flow.
• Net lines of code decreases by 1.
• Max indentation level decreases by 1.

I would argue that it also reads more like native language. “For each call to testConnReadNonzeroAndEOF that returns an error for a delay less than 64ms, multiply the delay by 2. If the final attempt returns an error, fail the test.”

You could also write the loop as:

delay := time.Millisecond
for err := testConnReadNonzeroAndEOF(t, delay); err != nil {
	if delay >= 64 * time.Millisecond {
		t.Fatal(err)
	}
	delay *= 2
}

Which would read as, “For each unsuccessful attempt at testConnReadNonzeroAndEOF, fail the test if the delay was at least 64ms, otherwise double the delay.”

[bcmills]

See also https://lwn.net/Articles/557073/:

This issue of expressions excluding assignments has clearly not gone unnoticed by the Go designers. If we look at the if and switch statements, we see that, while they can be given a simple expression, they can also be given a simple statement as well, such as:

	if i := strings.Index(s, ":"); i >= 0 {

which includes both an assignment and a test. This would work quite nicely for the readLine loop:

	while line, ok := file.readLine(); ok {

[hooluupog]

IMHO,if this proposal were accepted, #21263 should also be reconsidered. FYI,Swift has already implemented these features in 3.0(SE-0004, SE-0007).

[wesalvaro]

Why can't range (or range's replacement) return one or two things like checking for an item in a map? I rarely, if ever, use the index when using range.

for item := range items {}
for item, i := range items {}
for item : items {}
for item, i : items {}

Python and JavaScript both do this more "ergonomically":

for item in items: pass
for i, item in enumerate(items): pass
for i in range(len(items)): pass

items.forEach(item => {})
items.forEach((item, i) => {});

[bcmills]

@wesalvaro

Why can't range (or range's replacement) return one or two things like checking for an item in a map?

The first value in a map lookup is always the element, because the caller already has the key. The second value in a map lookup is the present boolean, which in a range loop would always be true. While it's tempting to make an argument from consistency, it seems to me that range and map lookups are inherently different operations.

(That's part of why I find the := range syntax awkward: x := m[k] and x := range m are very different operations, despite the similar syntax.)

I rarely, if ever, use the index when using range.

Yeah, I pretty much never use it either.

I can see two (mutually contradictory) arguments:

1. Loops over maps should be consistent with loops over slices and arrays.
2. Loops over single items should iterate over the most commonly-useful part.

Argument (1) favors using indices for both (or elements for both), whereas argument (2) favors indices for maps and elements for slices and arrays. Personally, I am partial to argument (2), as are you. But there are enough folks who favor argument (1) that there isn't a consensus that is obvious to me. So for this proposal I chose argument (3):

3. There is no choice that is “obvious” to everyone, so always be explicit.

I think ~everybody understands that when there are two variables they correspond to the index and element, respectively, and the leading _, is only three extra characters. That seems a small price to pay to avoid ambiguity.