Skip to content

Kind v0.8.x - K8s v1.11.10 Node Image Errors on Cluster Creation #1565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
bwagner5 opened this issue May 5, 2020 · 5 comments
Closed

Kind v0.8.x - K8s v1.11.10 Node Image Errors on Cluster Creation #1565

bwagner5 opened this issue May 5, 2020 · 5 comments
Assignees
@BenTheElder
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@bwagner5
Copy link

bwagner5 commented May 5, 2020

What happened:
Cluster provisioning when using the kind provided K8s v1.11.10 node images fails on Kind version v0.8.1

What you expected to happen:
Expected successful provisioning of a kind k8s cluster for v1.11.10 (all other K8s versions work with the provided images)

How to reproduce it (as minimally and precisely as possible):

$ kind create cluster --name nth-test-68e3cc35 --image kindest/node:v1.11.10@sha256:74c8740710649a3abb169e7f348312deff88fc97d74cfb874c5095ab3866bb42 --config kind-two-node-cluster.yaml --kubeconfig build/tmp-nth-test-68e3cc35/kubeconfig --retain

Anything else we need to know?:

$ kind create cluster --name nth-test-68e3cc35 --image kindest/node:v1.11.10@sha256:74c8740710649a3abb169e7f348312deff88fc97d74cfb874c5095ab3866bb42 --config kind-two-node-cluster.yaml --kubeconfig build/tmp-nth-test-68e3cc35/kubeconfig --retain
Creating cluster "nth-test-68e3cc35" ...
 ✓ Ensuring node image (kindest/node:v1.11.10) 🖼
 ✓ Preparing nodes 📦 📦
 ✓ Writing configuration 📜
 ✗ Starting control-plane 🕹️
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged nth-test-68e3cc35-control-plane kubeadm init --ignore-preflight-errors=all --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I0505 15:37:34.907857     166 masterconfig.go:113] loading configuration from the given file
I0505 15:37:34.910748     166 feature_gate.go:230] feature gates: &{map[]}
I0505 15:37:34.910843     166 init.go:250] [init] validating feature gates
[init] using Kubernetes version: v1.11.10
[preflight] running pre-flight checks
I0505 15:37:34.910912     166 checks.go:581] validating kubernetes and kubeadm version
I0505 15:37:34.911106     166 checks.go:179] validating if the firewall is enabled and active
I0505 15:37:34.918022     166 checks.go:216] validating availability of port 6443
I0505 15:37:34.918277     166 checks.go:216] validating availability of port 10251
I0505 15:37:34.918394     166 checks.go:216] validating availability of port 10252
I0505 15:37:34.918647     166 checks.go:291] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I0505 15:37:34.918915     166 checks.go:291] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I0505 15:37:34.918969     166 checks.go:291] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I0505 15:37:34.918978     166 checks.go:291] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I0505 15:37:34.919020     166 checks.go:438] validating if the connectivity type is via proxy or direct
I0505 15:37:34.919054     166 checks.go:474] validating http connectivity to first IP address in the CIDR
I0505 15:37:34.919090     166 checks.go:474] validating http connectivity to first IP address in the CIDR
I0505 15:37:34.919130     166 checks.go:138] validating if the service is enabled and active
	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
	[WARNING Service-Docker]: docker service is not active, please run 'systemctl start docker.service'
I0505 15:37:34.928593     166 checks.go:340] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
	[WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
I0505 15:37:34.928614     166 checks.go:340] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0505 15:37:34.928709     166 checks.go:653] validating whether swap is enabled or not
	[WARNING Swap]: running with swap on is not supported. Please disable swap
I0505 15:37:34.929045     166 checks.go:381] validating the presence of executable crictl
I0505 15:37:34.929122     166 checks.go:381] validating the presence of executable ip
I0505 15:37:34.929178     166 checks.go:381] validating the presence of executable iptables
I0505 15:37:34.929197     166 checks.go:381] validating the presence of executable mount
I0505 15:37:34.929208     166 checks.go:381] validating the presence of executable nsenter
I0505 15:37:34.929321     166 checks.go:381] validating the presence of executable ebtables
I0505 15:37:34.929422     166 checks.go:381] validating the presence of executable ethtool
I0505 15:37:34.929455     166 checks.go:381] validating the presence of executable socat
I0505 15:37:34.929537     166 checks.go:381] validating the presence of executable tc
I0505 15:37:34.929563     166 checks.go:381] validating the presence of executable touch
I0505 15:37:34.929581     166 checks.go:523] running all checks
I0505 15:37:34.931021     166 kernel_validator.go:81] Validating kernel version
I0505 15:37:34.931174     166 kernel_validator.go:96] Validating kernel config
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 4.19.76-linuxkit
CONFIG_NAMESPACES: enabled
CONFIG_NET_NS: enabled
CONFIG_PID_NS: enabled
CONFIG_IPC_NS: enabled
CONFIG_UTS_NS: enabled
CONFIG_CGROUPS: enabled
CONFIG_CGROUP_CPUACCT: enabled
CONFIG_CGROUP_DEVICE: enabled
CONFIG_CGROUP_FREEZER: enabled
CONFIG_CGROUP_SCHED: enabled
CONFIG_CPUSETS: enabled
CONFIG_MEMCG: enabled
CONFIG_INET: enabled
CONFIG_EXT4_FS: enabled
CONFIG_PROC_FS: enabled
CONFIG_NETFILTER_XT_TARGET_REDIRECT: enabled
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled
CONFIG_OVERLAY_FS: enabled
CONFIG_AUFS_FS: not set - Required for aufs.
CONFIG_BLK_DEV_DM: enabled
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
	[WARNING SystemVerification]: failed to get docker info: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
I0505 15:37:34.935173     166 checks.go:411] checking whether the given node name is reachable using net.LookupHost
I0505 15:37:34.936392     166 checks.go:622] validating kubelet version
I0505 15:37:35.050300     166 checks.go:138] validating if the service is enabled and active
I0505 15:37:35.058544     166 checks.go:216] validating availability of port 10250
I0505 15:37:35.058650     166 checks.go:216] validating availability of port 2379
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
I0505 15:37:35.058790     166 checks.go:253] validating the existence and emptiness of directory /var/lib/etcd
`docker` is required when docker is the container runtime and the kubelet is not running: exec: "docker": executable file not found in $PATH

$ cat kind-two-node-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
    - |
      apiVersion: kubeadm.k8s.io/v1beta2
      kind: ClusterConfiguration
      metadata:
        name: config
      apiServer:
        extraArgs:
          "enable-admission-plugins": "NodeRestriction,PodSecurityPolicy"
- role: worker%

Environment:

  • kind version: (use kind version):
$ kind version
kind v0.8.1 go1.14.2 darwin/amd64
  • Kubernetes version: (use kubectl version):
kind-worker-node $ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"7a578febe155a7366767abce40d8a16795a96371", GitTreeState:"clean", BuildDate:"2020-05-01T03:01:03Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
  • Docker version: (use docker info):
docker info
Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)
  app: Docker Application (Docker Inc., v0.8.0)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.778GiB
 Name: docker-desktop
 ID: TO3B:GCJF:M7JQ:NVVW:KOQ3:756K:NIBR:UEIZ:UR2M:MFO3:SXEX:Q3Z6
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 52
  Goroutines: 63
  System Time: 2020-05-05T15:43:54.827453043Z
  EventsListeners: 3
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
  • OS (e.g. from /etc/os-release):
    Mac OS X Catalina 10.15.4
$ uname -a
Darwin imac.lan 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64
@bwagner5 bwagner5 added the kind/bug Categorizes issue or PR as related to a bug. label May 5, 2020
@bwagner5
Copy link
Author

bwagner5 commented May 5, 2020 •
edited
Loading

I also tried building the image for k8s v1.11.10 from commit "7a578febe155a7366767abce40d8a16795a96371". That also failed in the same way as the provided dockerhub image.

Also, it is failing travis CI build (although logging is disabled on travis). I've reran the 1.11.10 build at least 6 times, so I'm fairly sure it's the same issue happening locally. You can also see all the other k8s versions are working fine: https://travis-ci.org/github/aws/aws-node-termination-handler/builds/683154828

@bwagner5 bwagner5 mentioned this issue May 5, 2020
Merged
@BenTheElder
Copy link
Member

Honestly we may just yank this image. 1.11.X has been out of support from the rest of the community since 2019-05-01

https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md#114-and-older

@BenTheElder
Copy link
Member

had to track down #1569 since that affects our current testing / development efforts on kubernetes master, but I am going to take a look at this soon ..

@BenTheElder
Copy link
Member

I'm sorry this took so long to get to, I think this is a kubeadm bug. I'm going to drop the image from the release and drop 1.11.X at least going forward, as it's been out of community support for a full year now.

I'm going to discuss this with some of our other contributors but I think our support policy going forward is going to be something like "everything kubernetes supports plus the unreleased code plus 3 older versions", which would currently be: (1.18.X, 1.17.X, 1.16.X) + 1.19.0-alpha + (1.15.X, 1.14.X, 1.13.X).

@BenTheElder
Copy link
Member

confirmed that 1.12.10 works. I've updated the release notes.

@BenTheElder BenTheElder self-assigned this May 28, 2020
kingdonb pushed a commit to fluxcd/helm-operator that referenced this issue Jun 23, 2021
@yebyen
One cannot test k8s 1.11 with kind anymore
a2708ff 
kubernetes-sigs/kind#1565

We will have to settle for testing with Kubernetes 1.12, but I will
leave the Chart.yaml requirement at 1.11 since there's no reason to
believe we have broken any compatibility with this release.

Signed-off-by: Kingdon Barrett <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BenTheElder BenTheElder

Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@BenTheElder @bwagner5