kaniko hangs for minutes doing nothing when destination is insecure http registry

[arve0]

Actual behavior
Kaniko hangs for minutes twice:

1. In beginning, after logline Adding /var/run to initialIgnoreList, about two and a half minute.
2. In end, after logline Pushing image to registry.default/arve0/kaniko-repro, 13 minutes.

Expected behavior
Kaniko should not try HTTPS without timeout when given --insecure-registry parameter, such that it does not hang for minutes doing nothing.

To Reproduce
Reproduced on Digital Ocean and Linode managed kubernetes.

Steps to reproduce the behavior:

1. Deploy a local registry HTTP registry, for example https://github.com/arve0/kaniko-repro/blob/master/registry.yaml
2. Start kaniko:

kubectl run kaniko --restart=Never --image=gcr.io/kaniko-project/executor:debug -- \
  --log-format=text \
  --verbosity=trace \
  --log-timestamp \
  --context=git://github.com/arve0/kaniko-repro \
  --insecure-registry=registry.default \
  --destination=registry.default/arve0/kaniko-repro \
  --force # force is required on linode, as runc detection does not work correctly

Alternative Reproduction
Installs registry and runs reproduction:

git clone https://github.com/arve0/kaniko-repro
cd kaniko-repro
./reproduce.sh

Additional Information

• Dockerfile, seem to be any Dockerfile. Has reproduced with

  FROM debian
  RUN echo hello > hello.txt 
  

• Build Context, no files needed. Reproduced with both mounted volume and git context. https://github.com/arve0/kaniko-repro

• Kaniko Image (fully qualified with digest), both :latest and :debug. Debug digest: sha256:fcccd2ab9f3892e33fc7f2e950c8e4fc665e7a4c66f6a9d70b300d7a2103592f.

• Trace logs:

  • linode complete without killing: https://raw.githubusercontent.com/arve0/kaniko-repro/master/linode-full.log
  • linode, kill -ABRT 1 on first hang: https://github.com/arve0/kaniko-repro/blob/master/linode-start-kill.log
  • linode, kill -ABRT 1 on second hang: https://raw.githubusercontent.com/arve0/kaniko-repro/master/linode-push-kill.log

• Running goroutines when hanging

  • first hang

  $ grep goroutine *start*log
  goroutine 0 [idle]:
  goroutine 1 [select]:
  goroutine 8 [select]:
  goroutine 9 [chan receive]:
  goroutine 34 [IO wait]:
  goroutine 40 [select]:
  goroutine 29 [IO wait]:
  

  • second hang

  $ grep goroutine *push*log
  goroutine 0 [idle]:
  goroutine 1 [select]:
  goroutine 8 [select]:
  goroutine 9 [chan receive]:
  goroutine 211 [select]:
  goroutine 205 [IO wait]:
  

What have I tried?
I've search issues after "hang", and found similar behavior described in #1287. Workaround in comment does not work. Also, my cluster have internet access, so not same issue I believe.

Adding HTTPS on port 443 to registry resolves the issue.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[arve0]

I've debugged some more, and it seems to only happen when using --insecure-registry=registry.default. Unable to reproduce when using same registry with ingress and cert-manager that has fully qualified domain name.

[arve0]

Reproduced with --insecure and --insecure-registry registry.default also.

Unable to reproduce if Service listens on port 443, for example:

apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: default
spec:
  selector:
    app: registry
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http
    - name: https
      protocol: TCP
      port: 443
      targetPort: http

Which of course is wrong, but hints to that kaniko tries to connect with HTTPS even when --insecure-registry is given. Or that I am using the flag wrong.

[tejal29]

we have seen this hanging issue in Skaffold's TestBuildKanikoInsecureRegistry reported here GoogleContainerTools/skaffold#7121

[agorgl]

Just bumped into this too!

[Enity]

spent hours looking for this. bump

[brezerk]

same here. bump