[Reappeared] Build and push to an insecure registry completes unsuccessfully

[NicklasWallgren]

Expected behavior
A Skaffold driven build should complete successfully even if the image is pushed to an insecure registry.

Actual behavior
Skaffold errors on building/pushing the new image.

FATA[0000] failed to build: tagging image: Get https://<registry>:32000/v2/: http: server gave HTTP response to HTTPS client

Works as expected in v0.35.0.

See #1732 for more information.

[balopat]

Hi @NicklasWallgren just making sure first - are you using insecureRegistries in either the CLI / skaffold.yaml / global config? https://skaffold.dev/search/?q=insecure%20registry Can you provide a bit more information on your setup/config? Are you using docker or not? If you are using the docker builder, are you using dockerCLI:true or not?

[dgageot]

@NicklasWallgren you are using Jib, right?

[dgageot]

@NicklasWallgren Can you try without artifact caching? It was activated by default in 0.36.0, I think. Something like skaffold build --cache-artifacts=false

If that works, I think it's because you now have to configure skaffold to handle insecure registries too. See https://skaffold.dev/docs/concepts/image_repositories/#insecure-image-registries.

Before 0.36.0, Skaffold would let Jib tag the images. Now, because of caching, Skaffold can sometimes retag images and it then requires to know that the registry is insecure.

[dgageot]

Maybe Skaffold could read the list of insecure registries from ~/.docker/deamon.json
See #3051

[NicklasWallgren]

I'm not using jib for this particular project, and I have specified the insecure registries in skaffold.yaml as well as in deamon.json.

I'll provide additional information tomorrow.

[NicklasWallgren]

@dgageot Worked fine without artifact caching enabled --cache-artifacts=false

[dgageot]

@NicklasWallgren Thanks! So there must be an issue. Do you have a small project we could use to reproduce?

[NicklasWallgren]

Hmm, it works as expected if I remove insecureRegistries from the skaffold.yaml.

@dgageot https://asciinema.org/a/f9bbw5cH94a5AoTCkaLNb2wAt

[NicklasWallgren]

@dgageot @balopat

It fails on https://github.com/google/go-containerregistry/blob/7683b4ee5f6150cb47a791309f781c522b95a58f/pkg/v1/remote/transport/transport.go#L43

Called from

skaffold/pkg/skaffold/docker/remote.go

Line 48 in 6236f7f

return remote.Write(targetRef, img, remote.WithAuthFromKeychain(authn.DefaultKeychain))

[NicklasWallgren]

reg.insecure is false in transport.go.
targetRef.Registry.insecure in AddRemoteTag is always false.

Possible solution. What do you think?

func AddRemoteTag(src, target string, insecureRegistries map[string]bool) error {
	logrus.Debugf("attempting to add tag %s to src %s", target, src)
	img, err := remoteImage(src, insecureRegistries)
	if err != nil {
		return errors.Wrap(err, "getting image")
	}

	targetRef, err := name.ParseReference(target, name.WeakValidation)
	if err != nil {
		return errors.Wrap(err, "getting target reference")
	}

	if IsInsecure(targetRef.Context().Registry.Name(), insecureRegistries) {
		targetRef, err = getInsecureRegistryImpl(target)
		if err != nil {
			logrus.Warnf("error getting insecure registry: %s\nremote references may not be retrieved", err.Error())
		}
	}

	return remote.Write(targetRef, img, remote.WithAuthFromKeychain(authn.DefaultKeychain))
}

@dgageot @balopat