existence of secret files are checked locally instead of remotely when using remote docker daemon with DOCKER_HOST

[danny-p93]

Description of the issue

I'm using an remote docker daemon (DOCKER_HOST is set). In my docker-compose.yml i have specified some secrets for use in my service. These secrets use a file that is only given on the remote system. Every time i use docker-compose on my local system it shows me a warning, that my service uses an undefined file. For my luck it is just a warning and the service starts and indeed uses the given files from the remote daemon.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

Output of docker version

Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:36 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.10
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.3
  Git commit:       9424aea
  Built:            Thu Aug  6 02:55:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc9
  GitCommit:        d736ef14f0288d6993a1845745d6756cfc9ddd5a
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

secrets:
  my_sec:
    file: /path/on/remote/system
services:
  my_ser:
    container_name: custom
    environment:
      SOMEENV: foo
    image: myimage:latest
    ports:
    - published: 1234
      target: 4321
    restart: always
    secrets:
    - source: my_sec
      target: /apps/project/path/in/container
    volumes:
    - /another/remote/path:/apps/project/path:rw
version: '3.7'

Steps to reproduce the issue

1. setup an remote docker daemon, that exposes the daemon port correctly
2. install docker-cli and docker-compose locally
3. specify DOCKER_HOST to the remote daemon
4. configure a local docker-compose.yml file, that includes a configuration for a secret, which uses a file, that only exists on the remote system
5. run the compose file

Observed result

i get a warning for the secret file
definition of remote volume works without warning

Expected result

i get a warning when the file is not present on the remote system and the local system should be ignored

Stacktrace / full error message

WARNING: Service "my_ser" uses an undefined secret file "/path/on/remote/system", the following file should be created "/path/on/remote/system"

Additional information

my remote system is a photon-os, my local system is ubuntu18.04 on WSL1
docker and docker-compose installed as documented

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[b3n3d1k7]

I have the exact same problem.
I'm running docker-compose on a remote DOCKER_HOST via ssh.

Both the local and remote system are Ubuntu 18.04.5 LTS.

docker-compose version:

docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

docker version

Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:46:01 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:44:13 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[bikol]

The same on docker version:

Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df35096d
 Built:             Fri Jun  4 08:14:39 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.4
  Git commit:       b0f5bc36fe
  Built:            Fri Jun  4 08:14:24 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.2
  GitCommit:        36cc874494a56a253cd181a1a685b44b58a2e34a.m
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker-compose version:

docker-compose version 1.29.2, build unknown
docker-py version: 5.0.0
CPython version: 3.9.5
OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021

Except the warning every thing works just fine.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Beanow]

(TL;DR, this is still an issue)

Ah yes. Frustrated with Swarm's missing features I gave compose another try. Not realizing compose's secrets are 'just bind volumes' and I would now have to Ansible over unencrypted secret/config files and come up with permission schemes.

In my case, coming from Swarm I was fully expecting:

secrets:
  example:
    file: ./local.secrets.conf

To submit the local file's contents over the Docker API, but instead it attempts to bind mount the absolute path it resolved on my local machine, which doesn't exist on the remote machine.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.

[pointlessone]

Just to keep the bot busy, this is till an issue.

[lylemoffitt]

Considering how Docker Swarm describes secrets as communicated over TLS, i was fully expecting that docker --context remote up would read a secret from a local file and send it over TLS to the remote context. Having it devolve to a bind mount makes it basically useless. Like the following are exactly the same, in this use case so why even bother with secrets?

secrets:
  example: 
    file: example.txt
services:
  foo:
    image: busybox
    command: cat /run/secrets/example 
    secrets:
      - example

services:
  foo:
    image: busybox
    command: cat /run/secrets/example 
    volumes:
      - ./example.txt:/run/secrets/example:ro

[Beanow]

Seems the stale bot went stale. https://github.com/apps/stale

Are we set up correctly to reopen this issue?

[alexey-sh]

Is there a way to open the issue, fix it, test it, and deploy it?