Docker build "FROM" Fails to search local images

[wtfbbqhax]

When building containers FROM another container image I've built locally, Docker build is failing to find the image and searching externally. A couple weeks ago, I saw this issue and found the problem was mitigated by disabling build kit (DOCKER_BUILDKIT=0) However, this no longer works.

Notably, this is a paradigm I've been following for years- I've searched all over dockers documentation and can not find any relevant changes documented.

This is what I'm seeing, The amd64/wtfbbqhax_runtime is just an alpine image that I have locally.

bash$ cat Dockerfile
FROM amd64/wtfbbqhax_runtime:7b2d02df45e4 as runtime
RUN apk add gdb
COPY cores/ /storage/cores
ENTRYPOINT []

bash$ docker images | grep amd64
amd64/wtfbbqhax_runtime          latest             7b2d02df45e4   3 hours ago   89.1MB

bash$ docker build -t wtfbbqhax-debug .
[+] Building 1.8s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 196B                                                                                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                                           0.0s
 => ERROR [internal] load metadata for docker.io/amd64/wtfbbqhax_runtime:7b2d02df45e4                                                                                                                                                                             1.6s
------
 > [internal] load metadata for docker.io/amd64/wtfbbqhax_runtime:7b2d02df45e4:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, 
repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[vemek]

I'm also seeing this. Dockerfiles that build on non-Mac hosts are failing to find local images on Mac. This is also the case for COPY --from. In my case, disabling buildkit mitigates the issue.

[stephen-turner]

Transferring to buildx repository.

[crazy-max]

@wtfbbqhax Looking at your diagnostic I would guess you're using the docker-container driver and buildx is used as an alias of docker build. What's the output of docker buildx ls and is there a builder alias defined in ~/.docker/config.json?

[wtfbbqhax]

@crazy-max Here's the outputs

bash$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  PLATFORMS
default * docker
  default default         running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

bash$ cat ~/.docker/config.json
{
  "auths" : {
    # Scrubbed
  },
  "stackOrchestrator" : "swarm",
  "credsStore" : "desktop"
}

[thaJeztah]

@wtfbbqhax I see your example shows

bash$ docker images | grep amd64
amd64/wtfbbqhax_runtime          latest             7b2d02df45e4   3 hours ago   89.1MB

(So the image has :latest tag), but you're referencing it as amd64/wtfbbqhax_runtime:7b2d02df45e (combination of image name and ID as tag)

Does it work if you reference it using the :latest tag?

[thaJeztah]

The format you're using looks to be the "shortid" reference, which was deprecated in docker 1.13; see docker/cli#753

[vemek]

In my case, I'm not using the shortid. Here's a minimal example:

First Dockerfile:

FROM debian:bullseye

RUN date > /root/date.txt

❯ docker build -t testbaseimage .
[+] Building 6.4s (6/6) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                    0.0s
 => => transferring dockerfile: 88B                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye                                                                                                                      2.5s
 => [1/2] FROM docker.io/library/debian:bullseye@sha256:4d6ab716de467aad58e91b1b720f0badd7478847ec7a18f66027d0f8a329a43c                                                                3.6s
 => => resolve docker.io/library/debian:bullseye@sha256:4d6ab716de467aad58e91b1b720f0badd7478847ec7a18f66027d0f8a329a43c                                                                0.0s
 => => sha256:4d6ab716de467aad58e91b1b720f0badd7478847ec7a18f66027d0f8a329a43c 1.85kB / 1.85kB                                                                                          0.0s
 => => sha256:21e79e173393a5c03e2dd910354affc1e6ea995aa3c88a18d40bae41760b3d4e 529B / 529B                                                                                              0.0s
 => => sha256:bf33d9a9dfb709a9a24218f6ea55bab3965d0029cdc2ce01214ad1a87040e530 1.48kB / 1.48kB                                                                                          0.0s
 => => sha256:1c47a423366578e5ce665d03788914bf0459485a627a27896fa9c5663ce55cdf 53.60MB / 53.60MB                                                                                        2.1s
 => => extracting sha256:1c47a423366578e5ce665d03788914bf0459485a627a27896fa9c5663ce55cdf                                                                                               1.3s
 => [2/2] RUN date > /root/date.txt                                                                                                                                                     0.2s
 => exporting to image                                                                                                                                                                  0.0s
 => => exporting layers                                                                                                                                                                 0.0s
 => => writing image sha256:fcc1089e6ed1548986b3c4a1a59feae838d7eadec573ccaa649e2f44e307b008                                                                                            0.0s
 => => naming to docker.io/library/testbaseimage                                                                                                                                        0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

❯ docker images testbaseimage
REPOSITORY      TAG       IMAGE ID       CREATED          SIZE
testbaseimage   latest    fcc1089e6ed1   39 seconds ago   118MB

Dockerfile that depends on that locally built image:

FROM testbaseimage:latest

RUN uname > /root/uname.txt

❯ docker build -t testimage .
[+] Building 1.5s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                    0.0s
 => => transferring dockerfile: 95B                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                         0.0s
 => ERROR [internal] load metadata for docker.io/library/testbaseimage:latest                                                                                                           1.4s
------
 > [internal] load metadata for docker.io/library/testbaseimage:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

With builkit: false in my config, this builds correctly.

Here's some more info in case it's useful:

❯ docker buildx ls
NAME/NODE        DRIVER/ENDPOINT  STATUS  PLATFORMS
desktop-linux    docker
  desktop-linux  desktop-linux    running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
myserver-remote   docker
  myserver-remote myserver-remote   running linux/amd64, linux/386
local-vm             docker-container
  localvm           local-vm             error during connect: Get "http://docker.example.com/v1.24/containers/buildx_buildkit_worm/json": command [ssh -p 8222 -- localhost docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host localhost port 8222: Connection refused

default * docker
  default default running linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

❯ cat ~/.docker/config.json
{
	"auths": {},
	"credsStore": "desktop"
}

[thaJeztah]

@vemek are you on an arm64 / apple m1 machine? Wondering if this is the same issue as described in moby/moby#42893

[vemek]

Ah, I am indeed. Sorry for the duplicate report here - I did not come across that issue when trying to figure this out.

[thaJeztah]

No worries! There's too many repositories that are "related in some way", so thanks for reporting the issue. There's definitely some issue there that needs to be looked into. This error should not happen (and if it happens, the error message should give a better clue why the image wasn't found (which - if my analysis on that ticket is correct - would be that it was not able to find a matching os/arch)

[thaJeztah]

For the image:<short-id> case, we should look why that still works with the classic builder; the <short-id> functionality was deprecated (and should be removed), but (I'd need to test it) possibly there's some legacy codepath that's hit, which makes it still work "partially". If possible, a warning (at least) should be printed to make users aware they're using an image reference format that's deprecated.

[wtfbbqhax]

@thaJeztah Ahh clarity at last. Yes it does work if I use :latest

[thaJeztah]

Thanks for confirming @wtfbbqhax ! (And glad you were able to resolve your case).

I'll check if it's possible to add a warning for this case to the classic builder, and (if possible) a clearer error message for BuildKit (because, as outlined in docker/cli#753, there are no plans to support the <short-id> syntax)

[thaJeztah]

Actually; trying to reproduce the <short-id> case on the classic builder, and that also seems to fail; what version of Docker were you using that it worked on? (curious)

Build an image:

DOCKER_BUILDKIT=0 docker build -t foobar -<<EOF
FROM alpine:latest
RUN echo hello > /world.txt
EOF

Find the short image ID:

docker image ls --filter reference=foobar
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
foobar       latest    ec1b305d37b4   13 seconds ago   5.6MB

Use that as reference;

DOCKER_BUILDKIT=0 docker build -t foobar -<<EOF
FROM foobar:ec1b305d37b4
RUN echo hello > /world.txt
EOF

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM foobar:ec1b305d37b4
pull access denied for foobar, repository does not exist or may require 'docker login': denied: requested access to the resource is denied