golang:1.12-alpine does not have layers for linux/arm64/v8

[moikot]

The result of running

docker run --rm weshigbee/manifest-tool inspect golang:1.12-alpine

...
4    Mfst Type: application/vnd.docker.distribution.manifest.v1+json
4       Digest: sha256:7cf1f7ccf392bd834eb91f02892f48992d3c2ba292c2198315a4637bb9454c30
4  Mfst Length: 6981
4     Platform:
4           -      OS: linux
4           -    Arch: arm64
4           - Variant: v8
4           - Feature:
4     # Layers: 0
...

Please note that the number of layers is zero. The same for golang:1.11-alpine

[wglambert]

Looks like an instance of docker-library/official-images#3835

The manifest is updated now

$ docker pull arm64v8/golang:1.12-alpine
1.12-alpine: Pulling from arm64v8/golang
3b00a3925ee4: Already exists 
7809c1a4c8e2: Pull complete 
8c00b1d46f44: Pull complete 
955cc90a48f7: Pull complete 
72f16051d572: Pull complete 
Digest: sha256:05f1d1242721f0042550fcc84f35cbe87f39ef5e6f75852d0608b92f4a2d1878
Status: Downloaded newer image for arm64v8/golang:1.12-alpine

[moikot]

The interesting thing is that I could pull 1.12 and 1.11 specifying arm64v8 prefix without a problem. But I can't do my builds using buildkit because it totally relies on the published manifests.
I've just checked again, no layers for 1.12 or 1.11 for arm64 😞 (1.10 is OK)

Here is the buildkit's error:

error: failed to solve: rpc error: code = Unknown desc = failed to copy: httpReaderSeeker: failed open: could not fetch content descriptor sha256:7cf1f7ccf392bd834eb91f02892f48992d3c2ba292c2198315a4637bb9454c30 (application/vnd.docker.distribution.manifest.v1+json) from remote: not found

[yosifkit]

After much digging. The manifest is technically fine. The item in question is a manifest v1:

4 Mfst Type: application/vnd.docker.distribution.manifest.v1+json

And works fine for docker pull. We have no control over what the hub gives us back when we push an image. The Hub gave us "500 Internal Server Error" twice while trying to push the alpine images of golang on the 7th of March. Doing an inspect on the pushed artifacts in arm64v8/golang shows that some pushed as v2 and some as v1. This makes me think that the Docker Hub was having issues and our docker client "helpfully" downgraded us to a v1 push by assuming it was an old registry.

cc @tianon, perhaps we need to somehow disable v1 pushing on our build jobs (this won't fix this now, but will prevent it in the future). edit: this would probably require custom patches to docker itself 😞

[tianon]

This is partly a bug in manifest-tool, so I've filed that: estesp/manifest-tool#74

I'm really not very keen on patching Docker for our builders; that's kind of heinous. 😞 😱

[tianon]

(Especially given that this was a blip in the Hub that caused these to be pushed in the first place, and docker pull can handle them just fine.)

[tonistiigi]

Could someone fix the current version of that golang:1.12-alpine image please. No official image created in the last 3 years should ever be schema1.

cc @dmcgowan For possible docker push side validation for this.

[tianon]

Is it still? There was a bump since then that should've pushed a new image that isn't schema1.

[tonistiigi]

@tianon Yes, golang:1.12-alpine is ok now. golang:1.12.0-alpine still produces the issue. So it's not as critical anymore(not sure if you usually would bump the previous versions as well in these cases). We should still find a solution where v1 variants could never happen though, even if some error appears in the process.

[tianon]

I'm kind of surprised the Hub still allowed a schema1 push 😅

[tonistiigi]

fwiw it seems likely to me that the configuration of the manifest list pointing to a v1 manifest will never be supported by containerd. Tracked in containerd/containerd#3100

[thaJeztah]

I'm kind of surprised the Hub still allowed a schema1 push 😅

@cowsrule any ideas? ^^

[cowsrule]

Also surprised, we should be blocking all v1 pushes, will followup internally. we also published this today: https://engineering.docker.com/2019/03/registry-v1-api-deprecation/