Wrong gateway is detected starting with Testcontainers 1.12.4

[ghostbuster91]

Hi,

I have a jenkins CI with a master node and two slaves. All tests are running smoothly on slaves but they fail once run on master. Master and slaves differs in terms of ubuntu version and docker version.
The master is:

API Version: 1.40
Operating System: Ubuntu 18.04.3 LTS

While the slaves are:

Server Version: 19.03.0
API Version: 1.40
Operating System: Ubuntu 16.04.6 LTS

I use testcontainers 1.12.4

Exceptions I am getting on master node:

13:11:08  [info]   org.testcontainers.containers.ContainerLaunchException: Container startup failed
13:11:08  [info]   at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:322)
13:11:08  [info]   at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302)
13:11:08  [info]   at com.dimafeng.testcontainers.SingleContainer.start(Container.scala:42)
13:11:08  [info]   ...
13:11:08  [info]   Cause: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageNameFuture=java.util.concurrent.CompletableFuture@687cf499[Completed normally], imagePullPolicy=DefaultPullPolicy(), dockerClient=LazyDockerClient.INSTANCE)
13:11:08  [info]   at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1265)
13:11:08  [info]   at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:600)
13:11:08  [info]   at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:311)
13:11:08  [info]   at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302)
13:11:08  [info]   at com.dimafeng.testcontainers.SingleContainer.start(Container.scala:42)
13:11:08  [info]   ...
13:11:08  [info]   Cause: java.lang.IllegalStateException: Can not connect to Ryuk
13:11:08  [info]   at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:150)
13:11:08  [info]   at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:135)
13:11:08  [info]   at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
13:11:08  [info]   at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
13:11:08  [info]   at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
13:11:08  [info]   at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
13:11:08  [info]   at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
13:11:08  [info]   at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:62)
13:11:08  [info]   at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:25)
13:11:08  [info]   at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)

and

12:11:37.140 [testcontainers-ryuk] WARN  o.t.u.ResourceReaper - Can not connect to Ryuk at localhost:32965
13:11:37  java.net.SocketException: Connection reset
13:11:37  	at java.net.SocketInputStream.read(SocketInputStream.java:210)
13:11:37  	at java.net.SocketInputStream.read(SocketInputStream.java:141)
13:11:37  	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
13:11:37  	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
13:11:37  	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
13:11:37  	at java.io.InputStreamReader.read(InputStreamReader.java:184)
13:11:37  	at java.io.BufferedReader.fill(BufferedReader.java:161)
13:11:37  	at java.io.BufferedReader.readLine(BufferedReader.java:324)
13:11:37  	at java.io.BufferedReader.readLine(BufferedReader.java:389)
13:11:37  	at org.testcontainers.utility.ResourceReaper$FilterRegistry.waitForAcknowledgment(ResourceReaper.java:406)
13:11:37  	at org.testcontainers.utility.ResourceReaper$FilterRegistry.register(ResourceReaper.java:402)
13:11:37  	at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:128)
13:11:37  	at java.lang.Thread.run(Thread.java:748)

Although when I call docker logs on master node

+ docker ps
CONTAINER ID        IMAGE                               COMMAND             CREATED             STATUS              PORTS                     NAMES
7b115c90863a        quay.io/testcontainers/ryuk:0.2.3   "/app"              32 seconds ago      Up 31 seconds       0.0.0.0:32965->8080/tcp   testcontainers-ryuk-75c3e431-6dd5-4faf-9b54-df86d395ff4e
++ docker ps -q
+ docker logs 7b115c90863a
2020/01/12 10:08:16 Starting on port 8080...

I can see that ryuk has started.

I assume that this is not a fault of ubuntu neither of docker version (mainly because I use similar versions on my local machine and it all works), but rather some specific configuration of master node. Unfortunately I have no idea how to debug it further, so any help would be appreciated.

Cheers!

[worldtiki]

I have the same issue. Downgrading to 1.12.3 seems to solve this.

[bsideup]

@worldtiki is it with 1.12.5 or 1.12.4?

[worldtiki]

@worldtiki is it with 1.12.5 or 1.12.4?

I have this issue with 1.12.5. I'm unable to test with 1.12.4 because of #2119

It's difficult for me to troubleshoot as I don't have access to the jenkins slaves where my tests run, but my guess is that they have different configurations of versions of docker as the error is intermittent (likely because it only affects some of the agents).

[bsideup]

@worldtiki you can test 1.12.4 just by calling DockerClientFactory.instance().client() somewhere before you start LocalStackContainer

[worldtiki]

So, quick update. With 1.12.4 it works ok (with the hack you suggested).

With 1.12.5 this is the error:

17:14:20.548 [main] INFO  o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
17:14:20.553 [main] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
17:14:20.769 [main] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is localhost
17:14:20.790 [main] INFO  o.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.09.9-ce
  API Version: 1.39
  Operating System: Amazon Linux 2
  Total Memory: 31367 MB
17:14:21.070 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:14:21.982 [testcontainers-ryuk] WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at localhost:53414
java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/java.net.Socket.connect(Socket.java:558)
	at java.base/java.net.Socket.<init>(Socket.java:454)
	at java.base/java.net.Socket.<init>(Socket.java:231)
	at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:114)
	at java.base/java.lang.Thread.run(Thread.java:834)

With 1.12.4

17:27:23.493 [main] INFO  o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
17:27:23.498 [main] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
17:27:23.986 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:25.065 [main] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
17:27:25.088 [main] INFO  o.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.09.9-ce
  API Version: 1.39
  Operating System: Amazon Linux 2
  Total Memory: 31367 MB
17:27:25.316 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:25.939 [main] INFO  o.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
        ?? Checking the system...
        ? Docker version should be at least 1.6.0
        ? Docker environment should have more than 2GB free disk space
17:27:26.466 [main] INFO  docker[localstack/localstack:0.9.4] - HOSTNAME_EXTERNAL environment variable set to 172.17.0.1 (to match host-routable address for container)
17:27:26.468 [main] INFO  docker[localstack/localstack:0.9.4] - Creating container for image: localstack/localstack:0.9.4
17:27:26.468 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: localstack/localstack:0.9.4, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:26.695 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:27.803 [main] INFO  docker[localstack/localstack:0.9.4] - Starting container with ID: 6075f529b3e1837404eda90ca0fcdc5be830abe59534f719ab71cc999646a94b

[ddierickx]

Can confirm that I have a similar regression with 1.12.5.

For the DockerComposeContainer, it did not use the correct Docker hostname when running inside docker (gitlab). Instead it used localhost. For a regular DockerContainer this worked fine and returned the correct hostname (IP address iso. localhost). Reverting to 1.12.4 fixed the issue for me.

[bsideup]

@ddierickx thanks, we will look at it. I wonder if this change is the one to blame:
#2151

[bsideup]

@ddierickx can you debug it on your side? Here are the steps:

1. Use 1.12.4
2. Obtain DockerClient by using DockerClientFactory.instance().client()
3. Run the following code:

var network = client.inspectNetworkCmd()
                         .withNetworkId("bridge")
                         .exec();
System.out.println(network);

var gateway = network
                         .getIpam()
                         .getConfig()
                         .stream()
                         .filter(it -> it.getGateway() != null)
                         .findAny()
                         .map(Network.Ipam.Config::getGateway)
                         .orElse("localhost");

assertEquals(DockerClientProviderStrategy.getDockerHostIpAddress(), gateway);

Please do share the log output if it fails

[ddierickx]

Changed the code a little bit since DockerClientProviderStrategy is not in 1.12.4, so I used EnvironmentAndSystemPropertyClientProviderStrategy instead.

        val client = DockerClientFactory.instance().client()
        val network = client.inspectNetworkCmd()
            .withNetworkId("bridge")
            .exec();
        println("Network: $network");

        val gateway = network
            .getIpam()
            .getConfig()
            .stream()
            .filter { it -> it.getGateway() != null }
            .findAny()
            .map { it.gateway }
            .orElse("localhost");

        System.out.println("Gateway: $gateway")

        EnvironmentAndSystemPropertyClientProviderStrategy()
            .getDockerHostIpAddress()
            .shouldBeEqualTo(gateway)

The output when running on our gitlab (with support for docker-in-docker by passing the docker socket) is:

Network: com.github.dockerjava.api.model.Network@4947a298[id=0742db67f2ff5b4706b8a3340248cd27e6d93c785304116e3c9b4911c1ea5a36
    ,name=bridge
    ,scope=local
    ,driver=bridge
    ,enableIPv6=false
    ,internal=false
    ,ipam=com.github.dockerjava.api.model.Network$Ipam@66efc28c[driver=default
    ,config=[com.github.dockerjava.api.model.Network$Ipam$Config@6ada782c]
    ,options=<null>]
    ,containers={b5de758269581432f830f7e78f8fe1e79ea5921eab554415f0c87102f25c90f9=com.github.dockerjava.api.model.Network$ContainerNetworkConfig@2fd37407[endpointId=afebcc4c3cf2e94b4309ccd2ee494ff73cdd9b7b0c4e44b360c93ef0f8084dc0
    ,macAddress=02:42:ac:11:00:02
    ,ipv4Address=172.17.0.2/16
    ,ipv6Address=]
    , edbef2cd0d08fca3f54e9149d0a3fdae91c76bafb1b8245bab101384ed450f7d=com.github.dockerjava.api.model.Network$ContainerNetworkConfig@5bf45007[endpointId=f20d40ccd6067864805411fd14c3d030f9be8fc8d5aed144b60c6540b9eed902
    ,macAddress=02:42:ac:11:00:03
    ,ipv4Address=172.17.0.3/16
    ,ipv6Address=]}
    ,options={com.docker.network.bridge.default_bridge=true
        , com.docker.network.bridge.enable_icc=true
        , com.docker.network.bridge.enable_ip_masquerade=true
        , com.docker.network.bridge.host_binding_ipv4=0.0.0.0
        , com.docker.network.bridge.name=docker0
        , com.docker.network.driver.mtu=1500}
        ,attachable=false
        ,labels={}]
Gateway: localhost
java.lang.AssertionError: Expected localhost actual 172.17.0.1.

Thanks for your help.

[NersesAM]

We have the same problem
1.12.5+ fails with WARN org.testcontainers.utility.ResourceReaper Can not connect to Ryuk at localhost:32768

INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
INFO  org.testcontainers.DockerClientFactory Docker host IP address is localhost
INFO org.testcontainers.DockerClientFactory Connected to docker: 
Server Version: 18.09.7
API Version: 1.39
Operating System: Ubuntu 18.04.3 LTS (containerized)

1.12.4

INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
INFO  org.testcontainers.DockerClientFactory Docker host IP address is 172.18.0.1
INFO org.testcontainers.DockerClientFactory Connected to docker: 
Server Version: 18.09.7
API Version: 1.39
Operating System: Ubuntu 18.04.3 LTS (containerized)

[bsideup]

@ddierickx thanks!

config=[com.github.dockerjava.api.model.Network$Ipam$Config@6ada782c]

it seems that docker-java did not add toString to Network$Ipam$Config.

Could you please try adding the following?

println("Network: ${JsonOutput.toJson(network)}");

[NersesAM]

@bsideup this is what I get on our CI

{
    "Id": "466fb33eef145fba9793a3bdbaef8b595df2267b5de806b4647485cd1ab827aa",
    "Name": "bridge",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "Internal": false,
    "IPAM": {
        "Driver": "default",
        "Config": [
            {
                "Subnet": "172.18.0.0/16",
                "IPRange": null,
                "Gateway": null,
                "NetworkID": null
            }
        ],
        "Options": null
    },
    "Containers": {
        "85e912f17b5501f6a683a70369bba46b1645a9a06c1e6e22388a7e051e918229": {
            "EndpointID": "<removed>",
            "MacAddress": "<removed>",
            "IPv4Address": "172.18.0.3/16",
            "IPv6Address": ""
        },
        "9391acd00344eaabf34f40f2c7116bde5f2209475ec8cdd968b03e345cbd6581": {
            "EndpointID": "<removed>",
            "MacAddress": "<removed>",
            "IPv4Address": "172.18.0.2/16",
            "IPv6Address": ""
        }
    },
    "Options": {
        "com.docker.network.bridge.default_bridge": "true",
        "com.docker.network.bridge.enable_icc": "true",
        "com.docker.network.bridge.enable_ip_masquerade": "true",
        "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
        "com.docker.network.bridge.name": "docker0",
        "com.docker.network.driver.mtu": "1500"
    },
    "Attachable": false,
    "Labels": {}
}