Build and push to an insecure registry completes unsuccessfully

[NicklasWallgren]

Expected behavior

A Skaffold driven build should complete successfully even if the image is pushed to an insecure registry.

Actual behavior

Skaffold errors on building/pushing the new image.

FATA[0005] build failed: building [<registry>:32000/<app>]: build artifact: getting image: Get https://<registry>:32000/v2/: http: server gave HTTP response to HTTPS client 

Works fine using jib directly.

gradle jib -Djib.to.image=<repository>:32000/<app>:latest

Information

• Skaffold version: 0.24

• ~/.docker/deamon.json

{
"insecure-registries" : [
    "<registry>:32000"
  ],
}

• Skaffold configuration

apiVersion: skaffold/v1beta2
kind: Config
build:
  artifacts:
    - image: <registry>:32000/<app>
      jibGradle: {}
      context: app

• Authentication

docker login <registry>:32000
Login Succeeded

• Jib configuration

jib {
   allowInsecureRegistries = true
}

[tejal29]

@NicklasWallgren i am new to the project and want to understand if this use case was supported in the previous versions of skaffold or this is a new feature request?

[NicklasWallgren]

I'm not sure really. I have tried versions 0.20 and 0.24. I thought it would work out-of-the-box since both jib and docker daemon are configured to allow insecure registries.

I'm using a simple microk8s registry.

[tejal29]

ok. Thanks. @balopat might have better idea.

[briandealwis]

I hit this too with Jib trying to push to a registry hosted within the cluster with k3s.io. #1705 hit the same issue with Kaniko.

[nkubala]

@NicklasWallgren thanks for the issue and @briandealwis thanks for linking the other issue. this is happening because of the way we retrieve the remote image reference once we've pushed an image with skaffold to retrieve the remote digest: since we're not telling go-containerregistry that our registry is insecure, it assumes it's not and defaults to https, which fails.

the quick and easy fix to this would be to tell skaffold that we're insecure before it tries to retrieve the remote image reference, but I'm not a fan of requiring the user to tell skaffold that their registry is insecure (through a flag or skaffold.yaml option or something else). I'm looking into ways we might be able to infer that without having to explicitly pass it in as a flag or option: maybe reading the ~/.docker/daemon.json, or finding another way to have docker tell us about insecure registries.

summoning @jonjohnsonjr, maybe you have a better idea?

[jonjohnsonjr]

There are some heuristics for matching insecure registries based on hostname and IP. If you can control the registry name, you can signal to go-containerregistry that you want it to be insecure by using the .local tld. I'd certainly be interested in supporting all the docker-y ways to override the insecure registry setting, I just haven't had time to plumb that through everywhere. I know there's daemon.json and a flag and an environment variable, at least. There's probably other methods as well.

PRs welcome 😄

[briandealwis]

Jib currently requires the user to explicitly allow using insecure registries (non-https).

It looks like the dockerd defaults to allowing insecure registries only for 127.0.0.0/8:

Local registries, whose IP address falls in the 127.0.0.0/8 range, are automatically marked as insecure as of Docker 1.3.2. It is not recommended to rely on this, as it may change in the future.

[NicklasWallgren]

@balopat Can we reopen this issue, since the PR was reverted?

[nkubala]

@NicklasWallgren got the fix merged today. we'll be releasing it tomorrow, but if you want to build from master and try it out let us know!

[NicklasWallgren]

@nkubala Works great, thanks :)

[NicklasWallgren]

This issue has reappeared in v0.36.0+

skaffold run -p dev
...
FATA[0000] failed to build: tagging image: Get https://<registry-url>:32000/v2/: http: server gave HTTP response to HTTPS client