go.mod:incorrect pacakge dependency in etcd 3.3 without go.mod

[cfc4n]

shell:

cfc4n@cnxct-mt:~/Project/golang $mkdir etcd_3.3_mod
cfc4n@cnxct-mt:~/Project/golang $cd etcd_3.3_mod
cfc4n@cnxct-mt:~/Project/golang $touch main.go

and type code with:

package main

import (
	"github.com/coreos/etcd/clientv3" //equal to go.etcd.io/etcd/clientv3
	"time"
)

func main() {
	cfg := clientv3.Config{
		Endpoints:        []string{"https://localhost:2379"},
		DialTimeout:      time.Second * 5,
	}

	_,e := clientv3.New(cfg)
	if e != nil {
		panic(e)
	}
}

condition 1

cfc4n@cnxct-mt:~/Project/golang/etcd_3.3_mod $go mod init etcd_3.3_mod
go: creating new go.mod: module etcd_3.3_mod
cfc4n@cnxct-mt:~/Project/golang/etcd_3.3_mod $go build -o test
go: finding github.com/coreos/pkg latest
go: finding github.com/coreos/go-systemd latest
# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
../../../../gopath/pkg/mod/github.com/coreos/etcd@v3.3.22+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
../../../../gopath/pkg/mod/github.com/coreos/etcd@v3.3.22+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
../../../../gopath/pkg/mod/github.com/coreos/etcd@v3.3.22+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
../../../../gopath/pkg/mod/github.com/coreos/etcd@v3.3.22+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

condition 2

error logs:

cfc4n@cnxct-mt:~/Project/golang/etcd_3.3_mod $go mod init etcd_3.3_mod
go: creating new go.mod: module etcd_3.3_mod
cfc4n@cnxct-mt:~/Project/golang/etcd_3.3_mod $go get go.etcd.io/etcd
go: finding github.com/coreos/pkg latest
go: finding golang.org/x/time latest
go: finding golang.org/x/net latest
go: finding github.com/golang/groupcache latest
go: finding github.com/coreos/go-systemd latest
go: finding github.com/tmc/grpc-websocket-proxy latest
go: finding github.com/xiang90/probing latest
go: finding google.golang.org/genproto latest
go: finding golang.org/x/crypto latest
go: github.com/coreos/bbolt@v1.3.5: parsing go.mod: unexpected module path "go.etcd.io/bbolt"
go: error loading module requirements

rootcause

There is not go.mod in https://github.com/etcd-io/etcd/tree/v3.3.22 . via https://pkg.go.dev/go.etcd.io/etcd/?tab=doc .
go mod will get lastest version package default.

package	ver in client used clientv3 lib	ver in etcd
github.com/coreos/etcd	v3.3.22+incompatible // indirect
github.com/coreos/go-semver	v0.3.0 // indirect	v0.2.0
github.com/coreos/go-systemd	v0.0.0-20191104093116-d3cd4ed1dbcf // indirect	v0.0.0-20190620071333-e64a0ec8b42a
github.com/coreos/pkg	v0.0.0-20180928190104-399ea9e2e55f // indirect	v0.0.0-20180108230652-97fdf19511ea
github.com/gogo/protobuf	v1.3.1 // indirect	v1.2.1
github.com/google/uuid	v1.1.1 // indirect	v1.0.0
go.etcd.io/etcd	v3.3.22+incompatible // indirect
go.uber.org/zap	v1.15.0 // indirect	v1.10.0
google.golang.org/grpc	v1.30.0 // indirect	v1.23.0

If we build project in etcd repo ,it works. because go mod will translate package version from glide.lock with correct version.
but if used clientv3 sdk ,it will failed ,go mod will not translate package for indirect package .and error logs with this issue.

other issue

#12028
#12009
#11931
#11829
#11808
#11772
#11749
#11721
#11707
and other complain : https://colobu.com/2020/04/09/accidents-of-etcd-and-go-module/ 🤣

solution

• add go.mod and go.sum into etcd 3.3 repo for GO111MODULE=on ( build this project and go get go.etcd.io/etcd for other project )
• keep use glide.lock and glide.yaml for GO111MODULE=off

/cc @gyuho @mitake @spzala

[cfc4n]

test and verify:

exec cmd: go get go.etcd.io/etcd@cdc1c8f02

cfc4n@cnxct-mt:~/Project/golang/etcd_3.3_mod $cat go.mod
module etcd_3.3_mod

go 1.12

require go.etcd.io/etcd v0.0.0-20200625034214-cdc1c8f02ff1
cfc4n@cnxct-mt:~/Project/golang/etcd_3.3_mod $

It was fixed.

@gyuho Can you release a new version for this issue?
Must of people use 3.3.22 now .
ps: ChangeLog PR:#12079

[bozaro]

Still broken in etcd 3.4: #12109

[ptabor]

Shouldn't we be concerned that this change is introducing inconsistent approaches between branches:

A) release-3.2 -> legacy - no go.mod at all

B) release-3.3. -> was no go.mod (as 3.2), but this PR introduces >> module go.etcd.io/etcd
<< that is effectively violating go versioning rule [1] and creating misleading experience.

Moreover release-3.3. still using github.com/coreos/etcd package naming, so the module name
go.etcd.io/etcd is incompatible.

C) release-3.4 -> Is using module go.etcd.io/etcd violating the go versioning rules.

D) release-3.5 in particular with #11820 introduces:

• proper module name "go.etcd.io/etcd/v3"
• change of import paths (fix go.etcd.io/etcd versioned import path  #11820) such that they follow the convention.

State A) is proper legacy state (+incompatible) according to go modules:

# go get go.etcd.io/etcd@v3.2.22

go: go.etcd.io/etcd v3.2.22 => v3.2.22+incompatible

State D) is proper go.mod state.

# go get go.etcd.io/etcd/v3@master

works and resolves to

go.etcd.io/etcd/v3 v3.3.0-rc.0.0.20200701155350-15884e908549

The referenced 15884e9 is fresh commit.
The awkward v3.3.0 is due to: f7a395f that is the last tagging on the branch.
We might consider tagging this branch as v3.5.0-pre to mitigate this misleading resolution.

State B) & D) are not supported and causing problems for tools:

# go get go.etcd.io/etcd/v3@v3.4.9

go get go.etcd.io/etcd/v3@v3.4.9: go.etcd.io/etcd/v3@v3.4.9: invalid version: go.mod has non-.../v3 module path
"go.etcd.io/etcd" (and .../v3/go.mod does not exist) at revision v3.4.9

# go get go.etcd.io/etcd@v3.4.9

go get go.etcd.io/etcd@v3.4.9: go.etcd.io/etcd@v3.4.9: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v3

# go get go.etcd.io/etcd@v3.4.9

go get go.etcd.io/etcd@v3.4.9+incompatible: go.etcd.io/etcd@v3.4.9+incompatible: invalid version: +incompatible suffix not allowed: module contains a go.mod file, so semantic import versioning is required

---

Some workaround is to do:

# go get go.etcd.io/etcd@release-3.4

That leads to awkward binding of:

go: go.etcd.io/etcd release-3.4 => v0.5.0-alpha.5.0.20200626183007-99e893d28560
go: downloading go.etcd.io/etcd v0.5.0-alpha.5.0.20200626183007-99e893d28560

Options forward

A) Revert this change and preserve the supported legacy behavior of 3.2 on 3.3 branch.

B) Backport #11820 to 3.3. & 3.4 patch, changing the module name to: go.etcd.io/etcd/v3 and import paths.
Its a very big change for patch release (formally disallowed)

C) (might by combined with A) Accelerate release of 3.5.0 branch to give customers a proper go-mod compatible dependency.
Prior to 3.5.0 release I would consider moving etcdv3client to a separate go module, as majority of customers want to depend on the go client alone and not on whole etcd implementation.

[cfc4n]

About option A, If revert this change ,The problem still appears like tihs issue. some package dependency incorrect..

In release-3.3 ,the module name still was go.etcd.io/etcd in go.mod after run go get github.com/coreos/etcd/clientv3 without this PR.and same as with this PR . So I can't understand the difference between with them in release-3.3

by the way,I am also confused module name with go.etcd.io/etcd and go.etcd.io/etcd/v3.

@ptabor

[ptabor]

Have you tried playing with replace() [1] in your go.mod ?

Kubernetes is using:

replace (
github.com/coreos/bbolt => github.com/coreos/bbolt v1.3.2
github.com/coreos/etcd => github.com/coreos/etcd v3.3.10+incompatible
)

While in theory, following directives would be a cleaner representation:

replace (
github.com/coreos/bbolt => go.etcd.io/bbolt
github.com/coreos/etcd => go.etcd.io/etcd
)

[1] https://golang.org/ref/mod#tmp_15

[cfc4n]

Yeah,I know it. But use go mod replace is an extra operation. and many people do not know how to solve this who encounter this problem.

[xiegeo]

The proper way is to release a new v4 for go.mod. For any version > 1, pre and post go.mod are not compatible. Any workaround such as replace just breaks go get for that project and causes more problems down the road.

[ptabor]

George: I assume that the plan is that at the time etcd will release: go.etcd.io/etcd/v3 (that represents branch 3.5) that is proper v3 module that holds the package compatibility rule. So all the etcd/v3 packages will be compatible... they will just start late in the history. I don't think we can afford major version (v4) any time soon.

CFC4N: Extra steps are unfortunate, but they can be solved for 3.2 & 3.3 by proper documentation. I think current change grows the complexity.

[cfc4n]

CFC4N: Extra steps are unfortunate, but they can be solved for 3.2 & 3.3 by proper documentation. I think current change grows the complexity.

I don't think so. and I think this change makes it easy to use client v3.