cmd/go: trim go.sum without adding module requirements

[F21]

What version of Go are you using (go version)?

go version go1.11.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64" 
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go" 
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo" 
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/d/14-Workspace/go/github.com/F21/somerepo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2" 
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build348706889=/tmp/go-build -gno-record-gcc-switches"

What did you do?

1. Create go.mod: go mod init github.com/F21/somerepo.
2. Populate go.mod and go.sum by running go test ./....
3. One of the dependencies github.com/uber/jaeger-client-go conflicts with another dependency, so I bumped it to master: go get github.com/uber/jaeger-client-go@master.
4. There are now 2 entries for github.com/uber/jaeger-client-go in my go.sum:

github.com/uber/jaeger-client-go v2.15.0+incompatible h1:NP3qsSqNxh8VYr956ur1N/1C1PjvOJnJykCzcD5QHbk=
github.com/uber/jaeger-client-go v2.15.0+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk=
github.com/uber/jaeger-client-go v2.15.1-0.20190116124224-6733ee486c78+incompatible h1:b2A4zBpOpzqZoSlqlJ3OOe0yjd577TnQIHG1/KN7U8E=
github.com/uber/jaeger-client-go v2.15.1-0.20190116124224-6733ee486c78+incompatible/go.mod h1:WVhlPFC8FDjOFMMWRy2pZqQJSXxYSwNYOkTr/Z6d3Kk=

The github.com/uber/jaeger-client-go v2.15.0+incompatible entries are unnecessary as the project is now pinned to github.com/uber/jaeger-client-go v2.15.1-0.20190116124224-6733ee486c78+incompatible (current master).

In #26381, the ability to trim go.sum was added to go mod tidy. However, go mod tidy is something I avoid as it pulls in all test dependencies for all my dependencies and their dependencies for all platforms. Unfortunately, go mod tidy has burnt me quite a few times and causes a lot more problems than it solves.
See:

• https://groups.google.com/forum/#!topic/golang-nuts/5otdvVra0xg
• cmd/go: mod tidy pulls in too many dependencies #27920

What did you expect to see?

I'd like to be able to trim unneeded entries from go.sum without having to run go mod tidy as its current behavior causes a lot more problems than it solves.

What did you see instead?

N/A.

[ghasemloo]

It would be nice to have a way to cleanup go.sum from stuff that are not relevant for the building the module and only keep the minimal required.

The current behavior adds hashes for all the stuff in one's go root which one might not want to expose to others (privacy) when pushing the module to a repo. Right now I need to manually cleanup the go.sum file before pushing it to the repo.

relevant bug: #26381

[qbiqing]

Any update on this? Am I right in understanding that there is still no way to trim hashes from go.sum which belong to packages that have been removed from go.mod?

[jayconrod]

@qbiqing go mod tidy does that, in addition to adding missing requirements in go.mod and removing unnecessary requirements.

This issue is about having separate functionality to trim sums from go.sum that go mod tidy would add, including test dependencies of dependencies and modules that only provide packages on platforms that aren't supported by the module author.

Since this is a very narrow case, and since having additional hashes in go.sum doesn't have any build or run-time cost, we haven't prioritized it.

[jayconrod]

@ghasemloo

The current behavior adds hashes for all the stuff in one's go root which one might not want to expose to others (privacy) when pushing the module to a repo. Right now I need to manually cleanup the go.sum file before pushing it to the repo.

go mod tidy doesn't add sums for anything outside the current module and its dependencies. After running go mod tidy, go.sum contains sums for modules providing packages needed to build packages in the main modules, as well as tests of those packages, on any platform. It doesn't depend on anything in GOROOT or GOPATH outside the main module, so there should be no privacy issue. go mod tidy has more info.

[ghasemloo]

I wrote some code to do this.
I will upload to a github project and share the link.

[ghasemloo]

here is the hacky way I did it:

gosumtrim/gosumtrim.go

// Binary gosumtrim trims the go.sum file based on the given go.mod file.
package main

import (
	"bufio"
	"flag"
	"fmt"
	"io"
	"os"
	"strings"

	"github.com/golang/glog"
)

var (
	modFile = flag.String("mod", "", "input go.mod file")
	sumFile = flag.String("sum", "", "input go.sum file")
	outFile = flag.String("out", "", "output go.sum file")
)

func main() {
	flag.Parse()

	modIn, err := os.Open(*modFile)
	if err != nil {
		glog.Exitf("Open(%q) failed: %v", *modFile, err)
	}
	sumIn, err := os.Open(*sumFile)
	if err != nil {
		glog.Exitf("Open(%q) failed: %v", *sumFile, err)
	}
	sumOut, err := os.Create(*outFile)
	if err != nil {
		glog.Exitf("Create(%q) failed: %v", *outFile, err)
	}

	if err := trim(modIn, sumIn, sumOut); err != nil {
		glog.Exitf("Trim() failed: %v", err)
	}
	if err := sumOut.Close(); err != nil {
		glog.Exitf("os.Close() fialed: %v", err)
	}
}

func trim(mod io.Reader, sum io.Reader, o io.Writer) error {
	m := bufio.NewReader(mod)
	s := bufio.NewReader(sum)

	// Read till we get to the require line.
	for {
		line, err := m.ReadString('\n')
		if err != nil {
			return fmt.Errorf("reading mod file failed: %v", err)
		}
		glog.Infof("line: %v", line)
		if line == "require (\n" {
			break
		}
	}

	// Process require lines and add dependencies into used map.
	used := make(map[string]bool)
	for {
		line, err := m.ReadString('\n')
		line = strings.Trim(line, "\n \t")
		if err != nil {
			return fmt.Errorf("reading input go.mod failed: %v", err)
		}
		glog.Infof("line: %v", line)
		if line == ")" {
			break
		}
		used[line] = true
	}

	glog.Infof("used: %v", used)

	// Filter sum->out, dropping what is not used in mod.
	for {
		line, err := s.ReadString('\n')
		if err == io.EOF {
			break
		}
		glog.Infof("line: %v", line)
		if err != nil {
			return fmt.Errorf("reading input go.sum failed: %v", err)
		}
		parts := strings.Split(strings.TrimSpace(line), " ")
		if len(parts) != 3 {
			return fmt.Errorf("expecting 3 parts per go.sum line, got %v", parts)
		}
		if !used[strings.Join(parts[:2], " ")] {
			continue
		}
		if _, err := o.Write([]byte(line)); err != nil {
			return fmt.Errorf("writing output go.sum failed: %v", err)
		}
	}
	return nil
}

gosumtrim/gosumtrim_test.go

package main

import (
	"bytes"
	"io"
	"io/ioutil"
	"os"
	"testing"

	"github.com/google/go-cmp/cmp"
)

const (
	root       = ".../gosumtrim/" // module root
	inModFile  = root + "testdata/go.mod.in"
	inSumFile  = root + "testdata/go.sum.in"
	outSumFile = root + "testdata/go.sum.out"
)

func TestTrim(t *testing.T) {
	mod := mustOpen(t, inModFile)
	sum := mustOpen(t, inSumFile)
	out := mustOpen(t, outSumFile)
	want, err := ioutil.ReadAll(out)
	if err != nil {
		t.Fatalf("ioutil.ReadAll(out) failed: %v", err)
	}

	b := bytes.NewBuffer(nil)
	if err := trim(mod, sum, b); err != nil {
		t.Fatalf("trime() failed: %v", err)
	}
	got := b.Bytes()

	t.Logf("got:\n%v", string(got))
	t.Logf("want:\n%v", string(want))

	if diff := cmp.Diff(string(got), string(want)); diff != "" {
		t.Fatalf("trime() returned:\n%s", diff)
	}
}

func mustOpen(t *testing.T, fn string) io.Reader {
	t.Helper()
	r, err := os.Open(fn)
	if err != nil {
		t.Fatalf("os.Open(%q) failed: %v", fn, err)
	}
	return r
}

[ghasemloo]

https://github.com/ghasemloo/gosumtrim