cmd/go: vendored dependencies lack language version annotations

[bcmills]

In writing #36875, I realized that we will have a bit of a problem with vendored dependencies if and when we start making incompatible changes to the language.

Specifically, the vendor directory is a flattened package tree, and it does not include go.mod files unless the vendored package dependencies happen to include a package at the root of the vendored module. Many modules (such as golang.org/x/tools) don't even have an importable package at the root, let alone one that someone would want to import.

Unfortunately, the go.mod files are currently the only place where we record the language version to use for those dependencies. (I don't know what language version we end up using for -mod=vendor builds today, but I don't see how it could possibly be the correct one in general.)

I can think of two possible fixes:

1. Extend the ## annotation comments that we added for cmd/go: automatically check and use vendored packages #33848 so that, for any module that provides one or more packages, we annotate the go version (if any) found in the dependency's original go.mod file. (Fortunately, we intentionally designed that part of the format to tolerate the addition of new annotations.)
   For example:

   # golang.org/x/text v0.3.3
   ## explicit, go 1.14
   golang.org/x/text/number
   

2. Explicitly copy in the go.mod files for each module that provides one or more packages, even if there is no corresponding package in that part of the vendor tree.

I have a slight preference for approach (1) but don't feel strongly about it.

(CC @ianlancetaylor @griesemer @jayconrod @matloob @mvdan)

[mvdan]

If anyone upgrades to a new Go version, and updates their go.mod to use that newer language version which happens to remove a language feature, could the build break until the author re-runs go mod vendor? If that's planned, we should make sure to mention it well in the changelog.

[bcmills]

@mvdan, ideally, a user changing the language version in their own go.mod file should not affect the language version used to compile their dependencies (regardless of vendoring), and therefore should not require any update to their vendor tree.

[mvdan]

Hm, maybe I misunderstood. I thought re-running go mod vendor with Go 1.15 could add a bunch of those extra ## lines, necessary to add the missing language version information for vendored dependencies.

[bcmills]

Re-running go mod vendor with Go 1.15 presumably would add some ## lines, but I don't think that should be conditioned on the go version declared in the main module's go.mod file.

To minimize churn in vendor/modules.txt, we could omit the version annotations for any module that specifies a version at least as old as the default (#36875). As long as those dependencies built with Go 1.14 (or whatever default version we choose), they should continue to build without re-vendoring.

[bcmills]

(The go version annotations probably do not need a consistency check like the one added in #33848, because they currently have no effect on the computed module dependency graph.)

[thepudds]

@bcmills

2. Explicitly copy in the go.mod files for each module that provides one or more packages, even if there is no corresponding package in that part of the vendor tree.

FWIW, I would suggest thinking a bit more about the possible virtues of your approach 2., including as things evolve in the future (e.g., perhaps another bit of information might get added to go.mod in the future).

Also, some small-ish chance that storing the current go.mod files might be a stepping stone towards storing old go.mod files as well, which might end up enabling more cmd/go functionality when in vendor mode (e.g., #30240 (comment) and next couple comments there), though I think storing the old go.mod files was not a preferred approach in the past.

[jayconrod]

I'd lean toward option (1). Seems simpler.

go.mod files affect module boundaries, so it would be hard to get this right. @bcmills mentioned in a meeting that commands run inside the vendor directory would get confused (a go.mod inside the vendor directory would define the main module, possibly the wrong main module). Tools and editors would have this problem, too.

[bcmills]

Seems like the vendor/modules.txt annotation is the way to go: that way we only have to load one file (vendor/modules.txt) to be able to build packages in vendor mode, rather than N different go.mod files, and the diffs are generally smaller, and we won't have to change the version logic if we happen to make go mod vendor start pruning out go.mod files (for example, in order to be less confusing for tools and editors).

[andybons]

@bcmills what are the next steps here?