cmd/go: support module-local install/run of tool dependencies

[myitcv]

What version of Go are you using (go version)?

go version go1.11 linux/amd64

What did you do?

Tied to install and run a tool dependency in a number of my modules (more detail below)

What did you expect to see?

A nice easy way to (install and) run a tool dependency.

What did you see instead?

I needed to set GOBIN and update PATH for each module.

Further detail (on What did you do?)

This issue builds on top of #25922 so if that changes shape in any significant way it may void what follows.

Per #27643 (comment), I think we need to make the workflow around using a module's tool dependencies easier. Let me try to explain by covering my "workflow".

The tools I use on a day-to-day basis fall into two categories:

1. Tools I want to be controlled by a project's go.mod
2. Tools that I need globally

Dealing with these in reverse order.

Category 2: I think it's clear that with Go 1.11, there is a "gap" here and that this is covered by #24250. Per the detail in that discussion, there are open questions on how to handle multiple versions, where the installed binaries should be put etc, but it all falls under that issue.

Category 1: by far the largest category of tools for me, made up largely of code generators that I use with go generate and the like. I absolutely want these to be version controlled. And I don't want to be using (via my PATH) a "global" install of such a tool, even if the version just happens to match at that point in time. But both go get and go install currently (i.e. Go 1.11) have a target of $GOPATH/bin (ignoring multi-element `GOPATH values for now).

Hence the workflow I have effectively adopted, building on #25922, is to create a module-local install target:

# create a new module
cd $(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello

# set GOBIN as a module-local install target
export GOBIN=$PWD/.bin

# update my PATH accordingly (I actually use https://github.com/cxreg/smartcd for this)
export PATH=$GOBIN:$PATH

# add a tool dependency (by definition, category 1 tool) following "best practice" laid out in 
# https://github.com/golang/go/issues/25922#issuecomment-412992431
cat <<EOD > tools.go
// +build tools

package tools

import (
        _ "golang.org/x/tools/cmd/stringer"
)
EOD

# install the tool
go install golang.org/x/tools/cmd/stringer

# verify we are using the module-local binary
which stringer

# which gives something like:
# /tmp/tmp.Hh0BNOF6k2/hello/.bin/stringer

As covered in #27643, one of the disconnects in Go 1.11 is that a go get or go install in a module context modifies the "local" go.mod but installs "globally". This is, as @bcmills put it, "weird". But is to my mind a gap in Go 1.11, just as there not being a "global" tool install is a gap (i.e. #24250).

Potential solutions

Just listing these here as a starting point:

• go run is a potential alternative to the "local" install here (and a very attractive one to my mind), but we need to find a way to address cmd/go: go run pkg is significantly slower than running built binary #25416.
• There could be some convention that a .bin/ directory, alongside a go.mod, is the target for go get and go install (of main packages) for "local" installs? But this wouldn't obviate the need for everyone to update their PATH and indeed .gitignore the .bin directory for every module they work on.
• ...

cc @bcmills @rogpeppe @mvdan

[bcmills]

(CC @rsc)

[bcmills]

The go run option seems pretty clean to me, and wouldn't expand the command surface at all.

[bcmills]

Another option might be some sort of go tool integration: go install -tool golang.org/x/tools/cmd/stringer and go tool stringer $pkg

[gregwebs]

Have you tried retool? I am successfully using that for all my tool installs.

[myitcv]

Have you tried retool? I am successfully using that for all my tool installs.

@gregwebs thanks for the link, yes that had come across my radar before.

I think we should avoid dependencies beyond the standard Go distribution where possible, particularly for, what some might consider, something as fundamental as this.

[mvdan]

I think this is especially an issue not to fix with third party tools. How would you ensure that the third party tool to install tools can be easily run by all developers? It creates a circular dependency that's entirely avoided if this mechanism is available in the Go toolchain.

[gregwebs]

Since you asked, the way I ensure it can be run by all developers is with which retool >/dev/null 2>&1 || go get github.com/twitchtv/retool before running retool do... Both of which are ran in a wrapper; in many projects this would be a (cluttered) Makefile for invoking make check, etc.

I agree with the sentiment though that a package manager should solve this issue. However,

• I do think it is essential to explore the design space with third party tools first.
• using just go modules by itself would make go the only programming language I know of that isn't developing a second utility to improve package management. This could be a great thing, but certainly the problem of making a second utility for package management easily run by all developers has been dealt with in other programming languages

There's another desire here that is somewhat related: running any command, not just a tool in the environment that contains tool installs, the packages, and other supported environment modifications. In the stack package manager, this is done with stack exec.

[myitcv]

I do think it is essential to explore the design space with third party tools first.

Agreed. This is very much the approach being followed by https://github.com/rogpeppe/gohack, too. Depending on the lessons learned, gohack may die, bits may be absorbed back into the go tool.... any number of possible outcomes.

Since you asked, the way I ensure it can be run by all developers is with which retool >/dev/null 2>&1 || go get github.com/twitchtv/retool before running retool do... Both of which are ran in a wrapper; in many projects this would be a (cluttered) Makefile for invoking make check, etc.

I think my main concern is simply that the guidance for every developer globally then becomes:

1. Install Go - follow the official docs
2. Install retool - following the docs on this third-party website

at which point you've "broken" one of the key benefits of Go. Not least because this combined guidance will not, by definition, be part of the official docs.

Hence why I'm keen that we get the functionality "interfaces" right here. It's also worth bearing in mind that modules are still officially experimental in 1.11, so there is still time.