cmd/compile: add GOEXPERIMENT=loopvar

[rsc]

@dr2chase has been working on an implementation of the for loop scoping change in discussion #56010. Before we get to a formal proposal, it would be good to make the code available for people to try in their own code. I propose we add the change behind a GOEXPERIMENT=loopvar setting, so that developers can judge the effect in their own code, including in the Go 1.21 release (by setting the environment variable and recompiling).

[gopherbot]

Change https://go.dev/cl/411904 mentions this issue: cmd/compile: experimental loop iterator capture semantics change

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[zigo101]

Could it be clarified more:

1. Does this apply to both for-range and for;;?
2. Does this ignore all go.mod files?

[dr2chase]

Applies to both for-range and for;;, using the fancy Javascript rewrite so that side-effects to the "unshared" iteration variable that occur during (*) the serial execution of the loop body are picked up by the next iteration of the loop. (*) any side effect observed at the completion of that iteration of the loop, whether serial, synchronized, or racy+lucky.

The current state of the CL is that it will ignore go.mod files, but there is also per-package control for turning it off (-gcflags=whatever_package=-d-loopvar=-1, on ...loopvar=1, or accepting whatever is the default for the compiler or build ...loopvar=0). My model/plan at this stage is that I want to test this on as much code as possible, and in practice so far it works fine, so the risk from applying the change to entire applications is pretty low. Involving module version will just get in the way of running this on lots of code. I don't want people to be conservative, I want them to test it.

That said, there is also a hashed-search tool for repeatedly running a failing application with the change enabled in different places to search for any place where the change is a problem. I'm still tweaking that, and it also interacts with what I think is a bug in how source positions are recorded for names in inlined functions, but the idea would be if you have a command some command that go builds then fails to run then you would type lvh-search some command that go builds then fails to run and in a relatively short time (fewer than 10 executions of the failing command, most likely) it will tell you exactly where the failure-causing loop is. Right now lvh-search is spelled gossahash -e loopvarhash, and it works like this:

gossahash -e loopvarhash go run .
Trying go args=[run .], env=[GOCOMPILEDEBUG=loopvarhash=1]
Trying go args=[run .], env=[GOCOMPILEDEBUG=loopvarhash=0]
go failed (5 distinct triggers): exit status 1
Trying go args=[run .], env=[GOCOMPILEDEBUG=loopvarhash=00]
go failed (3 distinct triggers): exit status 1
Trying go args=[run .], env=[GOCOMPILEDEBUG=loopvarhash=000]
go failed (2 distinct triggers): exit status 1
Trying go args=[run .], env=[GOCOMPILEDEBUG=loopvarhash=1000]
go failed (1 distinct triggers): exit status 1
Review GSHS_LAST_FAIL.0.log for failing run
FINISHED, suggest this command line for debugging:
GOCOMPILEDEBUG=loopvarhash=1000 go run .
Problem is at /blahblahblah/src/cmd/compile/internal/loopvar/testdata/inlines/main.go:27:11

I.e., I want people to go into this feeling lucky, but if they're not lucky, I want them to find the problem in a few minutes with little-to-no mental effort. It's binary search, so finding the problem in 2500 loops takes only twice as long as finding the problem in a program with 50 loops (the Go compiler itself has 10,000-ish loops, but only about 50 that potentially leak their iteration variable).

What I expect (based on testing within Google) is that if you have a problem, it will be in a unit test, because the usual case for this has been with parallel subtest execution inside a loop with a captured iteration variable. When this happens, the pre-change behavior is to only run the last test in the series, and if it passes, you might never notice that there was a problem. If there is a problem in the not-run tests, the new behavior will uncover it (and it has done this). The hash search is a good match for this, since go test builds, then runs, so if you have a failing test, it is a simple matter of lvh-search go test -run TestThatFails . and it should take you directly there without wasting your time wondering which loop it is or if it is somewhere in your non-test code.

One tricky bit is whether the package enable/disable will work across inlines; that will need to happen by modules if/when we make the change, and that CL is already written and tested. Writing this all down has rubber-ducked me into thinking that the answer is "yes, the choice should follow across inlines"; if someone is trying this and encounters a problem in some module that they import but not their own code (i.e., it is in the module cache, readonly), I don't want them to be forced to download that code, modify it, and edit their application's go.mod to replace the import with its local version just to finish the testing. If they can disable it in the broken package on the command line, and that disable follows inlines, then they can more easily finish testing.

The other "tool" (perhaps just a script) I am trying to put together would combine information about where the transformation was applied (-gcflags=whatever_package=-d=loopvar=2) with coverage information so that it's possible to know that every transformed loop was tested. The new coverage in 1.20 can be used on entire applications, so this extends to more than just testing (though of course all Go programmers obtain 100% code coverage from their unit tests, right?)

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group