kaniko build: pod already in terminal phase: Failed

[r0bj]

Expected behavior

Using example from examples/kaniko-local works as expected.

Actual behavior

Running examples/kaniko-local generates error (GCP project name redacted):

$ skaffold dev -v debug
INFO[0000] Skaffold &{Version:v0.18.0 ConfigVersion:skaffold/v1alpha5 GitVersion: GitCommit:34651689be78b2c6bcfbace5072b00b93661f895 GitTreeState:clean BuildDate:2018-11-09T09:29:27Z GoVersion:go1.11.2 Compiler:gc Platform:darwin/amd64}
INFO[0000] no config entry found for kube-context gke_my-proj_us-west1-a_c1
INFO[0000] Using kubectl context: gke_my-proj_us-west1-a_c1
INFO[0000] no config entry found for kube-context gke_my-proj_us-west1-a_c1
DEBU[0000] Using builder: kaniko
DEBU[0000] Checking base image gcr.io/google-appengine/golang for ONBUILD triggers.
DEBU[0004] Found dependencies for dockerfile: [main.go]
Starting build...
Creating kaniko secret [e2esecret]...
DEBU[0004] No pull secret specified. Checking for one in the cluster.
Building [gcr.io/my-proj/skaffold-example]...
Storing build context at /tmp/context-8cc7b3ae1355d7af093c8f7abeb0de7c.tar.gz
DEBU[0006] Checking base image gcr.io/google-appengine/golang for ONBUILD triggers.
DEBU[0006] Found dependencies for dockerfile: [main.go]
INFO[0006] Waiting for kaniko-vm9x6 to be scheduled
INFO[0006] Waiting for kaniko-vm9x6 to be initialized
DEBU[0008] Running command: [kubectl cp /tmp/context-8cc7b3ae1355d7af093c8f7abeb0de7c.tar.gz kaniko-vm9x6://tmp/context-8cc7b3ae1355d7af093c8f7abeb0de7c.tar.gz -c kaniko-init-container -n default]
DEBU[0016] Running command: [kubectl exec kaniko-vm9x6 -c kaniko-init-container -n default -- tar -xzf /tmp/context-8cc7b3ae1355d7af093c8f7abeb0de7c.tar.gz -C /kaniko/buildcontext]
DEBU[0019] Running command: [kubectl exec kaniko-vm9x6 -c kaniko-init-container -n default -- touch /tmp/complete]
INFO[0022] Waiting for kaniko-vm9x6 to be ready
DEBU[0022] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-vm9x6" is waiting to start: PodInitializing
DEBU[0024] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-vm9x6" is waiting to start: PodInitializing
INFO[0024] no config entry found for kube-context gke_my-proj_us-west1-a_c1
INFO[0024] Using kubectl context: gke_my-proj_us-west1-a_c1
INFO[0024] no config entry found for kube-context gke_my-proj_us-west1-a_c1
DEBU[0024] Using builder: kaniko
Cleaning up...
DEBU[0024] manifests apiVersion: v1
kind: Pod
metadata:
  name: getting-started-kaniko
spec:
  containers:
  - name: getting-started
    image: gcr.io/my-proj/skaffold-example
DEBU[0024] Running command: [kubectl --context gke_my-proj_us-west1-a_c1 delete --ignore-not-found=true -f -]
DEBU[0025] unable to get kaniko pod logs: pods "kaniko-vm9x6" not found
Cleanup complete in 1.228702838s
FATA[0026] exiting dev mode because the first build failed: building [gcr.io/my-proj/skaffold-example]: kaniko build for [gcr.io/my-proj/skaffold-example]: waiting for pod to complete: pod already in terminal phase: Failed

Information

• Skaffold version: v0.18.0
• Operating system: GKE 1.11.2-gke.18

[priyawadhwa]

Hey @r0bj I was able to repro this with v0.18.0, but it seems to be working with skaffold when built @ HEAD. Maybe we (somehow) fixed it?

v0.19.0 is scheduled for next week, hopefully this should be resolved with that release.

[nkubala]

@r0bj v0.19.0 just went live. Could you install the latest release and see if this issue is fixed?

[r0bj]

@nkubala I have similar issue with version v0.19.0:

$ skaffold dev -v debug
INFO[0000] Skaffold &{Version:v0.19.0 ConfigVersion:skaffold/v1beta1 GitVersion: GitCommit:9eb0dfc1bf634b97462c66b4dfb80e4cea378ade GitTreeState:clean BuildDate:2018-11-29T22:22:17Z GoVersion:go1.10.5 Compiler:gc Platform:darwin/amd64}
INFO[0000] no config entry found for kube-context gke_my-proj_us-west1-a_c1
INFO[0000] Using kubectl context: gke_my-proj_us-west1-a_c1
INFO[0000] no config entry found for kube-context gke_my-proj_us-west1-a_c1
DEBU[0000] Using builder: kaniko
DEBU[0000] Checking base image gcr.io/google-appengine/golang for ONBUILD triggers.
DEBU[0000] setting Docker user agent to skaffold-v0.19.0
DEBU[0001] Found dependencies for dockerfile: [main.go]
Starting build...
Creating kaniko secret [e2esecret]...
DEBU[0001] No pull secret specified. Checking for one in the cluster.
Building [gcr.io/my-proj/skaffold-example]...
Storing build context at /var/folders/tf/34fxml8d3jl2vp_nkzx_15_r0000gn/T/context-785ac60113746419139512f330024539.tar.gz
DEBU[0002] Checking base image gcr.io/google-appengine/golang for ONBUILD triggers.
DEBU[0002] Found dependencies for dockerfile: [main.go]
INFO[0002] Waiting for kaniko-r6dx4 to be scheduled
INFO[0002] Waiting for kaniko-r6dx4 to be initialized
DEBU[0004] Running command: [kubectl cp /var/folders/tf/34fxml8d3jl2vp_nkzx_15_r0000gn/T/context-785ac60113746419139512f330024539.tar.gz kaniko-r6dx4:/tmp/context-785ac60113746419139512f330024539.tar.gz -c kaniko-init-container -n ops]
DEBU[0009] Running command: [kubectl exec kaniko-r6dx4 -c kaniko-init-container -n ops -- tar -xzf /tmp/context-785ac60113746419139512f330024539.tar.gz -C /kaniko/buildcontext]
DEBU[0011] Running command: [kubectl exec kaniko-r6dx4 -c kaniko-init-container -n ops -- touch /tmp/complete]
INFO[0013] Waiting for kaniko-r6dx4 to be ready
DEBU[0013] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-r6dx4" is waiting to start: PodInitializing
DEBU[0015] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-r6dx4" is waiting to start: PodInitializing
DEBU[0016] unable to get kaniko pod logs: container "kaniko" in pod "kaniko-r6dx4" is waiting to start: PodInitializing
DEBU[0000] Getting source context from dir:///kaniko/buildcontext
DEBU[0000] Build context located at /kaniko/buildcontext
DEBU[0000] Copying file /kaniko/buildcontext/Dockerfile to /kaniko/Dockerfile
INFO[0000] Downloading base image gcr.io/google-appengine/golang
WARN[0000] Error while retrieving image from cache: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1
INFO[0000] Downloading base image gcr.io/google-appengine/golang
error building image: getting stage builder for stage 0: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1
INFO[0018] no config entry found for kube-context gke_my-proj_us-west1-a_c1
INFO[0018] Using kubectl context: gke_my-proj_us-west1-a_c1
INFO[0018] no config entry found for kube-context gke_my-proj_us-west1-a_c1
DEBU[0018] Using builder: kaniko
Cleaning up...
DEBU[0018] manifests apiVersion: v1
kind: Pod
metadata:
  name: getting-started-kaniko
  namespace: ops
spec:
  containers:
  - name: getting-started
    image: gcr.io/fandom-prod/skaffold-example
DEBU[0018] Running command: [kubectl --context gke_fandom-prod_us-west1-a_c1 delete --ignore-not-found=true -f -]
Cleanup complete in 8.057848277s
FATA[0026] exiting dev mode because first run failed: build failed: building [gcr.io/fandom-prod/skaffold-example]: kaniko build for [gcr.io/fandom-prod/skaffold-example]: waiting for pod to complete: pod already in terminal phase: Failed

[dgageot]

I can reproduce. Let me investigate

[dgageot]

No, In fact I can't repro. My error was unrelated.

[r0bj]

Log from kaniko container:

DEBU[0000] Getting source context from dir:///kaniko/buildcontext
DEBU[0000] Build context located at /kaniko/buildcontext
DEBU[0000] Copying file /kaniko/buildcontext/Dockerfile to /kaniko/Dockerfile
INFO[0000] Downloading base image gcr.io/google-appengine/golang
WARN[0000] Error while retrieving image from cache: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1
INFO[0000] Downloading base image gcr.io/google-appengine/golang
error building image: getting stage builder for stage 0: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1

[priyawadhwa]

@r0bj I still wasn't able to repro -- I'm guessing there could be a network connection issue in your cluster? It looks like kaniko is trying to download the base image (gcr.io/google-appengine/golang) but is failing to do so.

[r0bj]

@priyawadhwa I've tried on two different clusters, one on GKE and second on bare metal - both with the same results. I didn't notice any connectivity issues on those clusters.

Test with the same url which is failing:

# kubectl run -it --image alpine test --restart Never --rm /bin/sh
[...]
/ # curl -svo /dev/null https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io
/ # *   Trying 64.233.161.82...
* TCP_NODELAY set
* Connected to gcr.io (64.233.161.82) port 443 (#0)
[...]
> GET /v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull HTTP/2
> Host: gcr.io
> User-Agent: curl/7.61.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< docker-distribution-api-version: registry/2.0
< content-type: application/json
< date: Mon, 03 Dec 2018 19:01:22 GMT
< server: Docker Registry
< cache-control: private
< x-xss-protection: 1; mode=block
< x-frame-options: SAMEORIGIN
< alt-svc: quic=":443"; ma=2592000; v="44,43,39,35"
< accept-ranges: none
< vary: Accept-Encoding

[imroc]

Log from kaniko container:

DEBU[0000] Getting source context from dir:///kaniko/buildcontext
DEBU[0000] Build context located at /kaniko/buildcontext
DEBU[0000] Copying file /kaniko/buildcontext/Dockerfile to /kaniko/Dockerfile
INFO[0000] Downloading base image gcr.io/google-appengine/golang
WARN[0000] Error while retrieving image from cache: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1
INFO[0000] Downloading base image gcr.io/google-appengine/golang
error building image: getting stage builder for stage 0: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1

I got the same issue, this is my output:

INFO[0000] Downloading base image gcr.io/google-appengine/golang
WARN[0000] Error while retrieving image from cache: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1
INFO[0000] Downloading base image gcr.io/google-appengine/golang
error building image: getting stage builder for stage 0: Get https://gcr.io/v2/token?scope=repository%3Agoogle-appengine%2Fgolang%3Apull&service=gcr.io: exit status 1

[dgageot]

See failing integration test: https://source.cloud.google.com/results/invocations/aea79c06-1e63-44e8-a0bd-4363aa61d39a/targets/runtimes-common%2Fskaffold%2Fpresubmit/log

[priyawadhwa]

I'm wondering if this could be related to this bug in kaniko --

GoogleContainerTools/kaniko#613

If so, it could potentially be fixed by changing the base image to golang:1.10

[dgageot]

Which base image? Le lun. 18 mars 2019 à 21:13, priyawadhwa <notifications@github.com> a écrit :

…

I'm wondering if this could be related to this bug in kaniko -- GoogleContainerTools/kaniko#613 <GoogleContainerTools/kaniko#613> If so, it could potentially be fixed by changing the base image to golang:1.10 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1307 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAJXl7yNjohEAWQ6QJ-vstVzNOIimkxgks5vX_NqgaJpZM4Yt3SR> .

[priyawadhwa]

Actually I made a mistake, nevermind! From the logs above I thought kaniko was building upon gcr.io/google-appengine/golang, but it looks like in our integration tests the Dockerfile actually pulls golang:1.10.1-alpine3.7