fix: handle stopped containers more gracefully when reuse is enabled

[georgespalding]

What does this PR do?

Fixes a bug that causes testcontainers to hang for a really long time waiting on a stopped container when reuse is enabled.

The fix in short is:

When a container is being created with reuse enabled:

• Look among all containers, even the ones not running.
  • If a container with the expected name is found:
    • If it is not running:
      • Attempt to start it.
        • If the start fails, return an error

The alternative cases above should work same as before

Why is it important?

The way the library works today, if a reused container exists, but is not running, it freezes for a long time, which is a really bad developer experience.

This fix is not watertight (ses the docker pause scenario below), but instantly tells you that something is wrong, giving fast feedback.

Related issues

• Relates [Enhancement]: Reuse flag to start exited or paused containers  #2470

How to test this PR

Create a little program that runs a testcontainer:

package main

import (
	"context"
	"fmt"
	"github.com/testcontainers/testcontainers-go"
	"github.com/testcontainers/testcontainers-go/modules/postgres"
)

func main() {
	true := true
	containerName := "my-postgres"
	container, err := postgres.Run(context.Background(),
		"postgres:14-bookworm",
		postgres.BasicWaitStrategies(),
		testcontainers.CustomizeRequestOption(func(req *testcontainers.GenericContainerRequest) error {
			req.Name = containerName
			req.Reuse = true
			return nil
		}),
	)
	if err != nil {
		fmt.Println(fmt.Sprintf("run postgres testcontainer: %s", err.Error()))
	} else {
		fmt.Println(fmt.Sprintf("postgres testcontainer is running: %t", container.Container.IsRunning()))
	}
}

1. docker ps -> container is not there
2. go run ./ -> prints postgres testcontainer is running: true
3. docker ps -> container is running
4. docker stop my-postgres
5. go run ./ -> prints postgres testcontainer is running: true
6. docker pause my-postgres
7. go run ./ -> prints run postgres testcontainer: generic container: create container: start container my-postgres in state paused: container start: Error response from daemon: cannot start a paused container, try unpause instead

[netlify]

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	77bfbc2
🔍 Latest deploy log	https://app.netlify.com/sites/testcontainers-go/deploys/680b609406923c000871a107
😎 Deploy Preview	https://deploy-preview-3062--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[mdelapenya]

@georgespalding thanks for submitting this PR. I'm trying to reproduce it locally with this test:

func TestWithReuse(t *testing.T) {
	containerName := "my-postgres"

	opts := []testcontainers.ContainerCustomizer{
		postgres.BasicWaitStrategies(),
		testcontainers.CustomizeRequestOption(func(req *testcontainers.GenericContainerRequest) error {
			req.Name = containerName
			req.Reuse = true
			return nil
		}),
	}

	ctr, err := postgres.Run(context.Background(), "postgres:14-bookworm", opts...)
	if err != nil {
		t.Logf("run postgres testcontainer: %s", err.Error())
	} else {
		t.Logf("postgres testcontainer is running: %t", ctr.IsRunning())
	}

	require.NoError(t, err)
	require.NotNil(t, ctr)

	err = ctr.Stop(context.Background(), nil)
	require.NoError(t, err)

	ctr1, err := postgres.Run(context.Background(), "postgres:14-bookworm", opts...)
	if err != nil {
		t.Logf("run postgres testcontainer: %s", err.Error())
	} else {
		t.Logf("postgres testcontainer is running: %t", ctr1.IsRunning())
	}

	require.NoError(t, err)
	require.NotNil(t, ctr1)

	cli, err := testcontainers.NewDockerClientWithOpts(context.Background())
	require.NoError(t, err)
	require.NotNil(t, cli)

	err = cli.ContainerPause(context.Background(), ctr1.GetContainerID())
	require.NoError(t, err)

	ctr2, err := postgres.Run(context.Background(), "postgres:14-bookworm", opts...)
	if err != nil {
		t.Logf("run postgres testcontainer: %s", err.Error())
	} else {
		t.Logf("postgres testcontainer is running: %t", ctr2.IsRunning())
	}

	require.NoError(t, err)
	require.NotNil(t, ctr2)
}

This test fails for ctr2 to be run, and this is because it needs to be unpaused first, which is the error message I see in the description of the PR.

I tested it against main and your code with the same results, although I spotted a different bug for reused containers: the code to skip starting a container if Ryuk was disabled I think is incorrect, and should apply always, checking if the container is in the unpaused state too (cc/ @stevenh)

docker.go:L1367

-	if p.config.RyukDisabled {
-		// ryuk disabled, ensure the container we are about to reuse is running
-		if c.State != "running" {
-			// found container to reuse, but it is not in state running
-			if err := dc.Start(ctx); err != nil {
-				return dc, fmt.Errorf("start container %s in state %s: %w", req.Name, c.State, err)
-			}
-		}
-	}
+	// Ensure the container we are about to reuse is running,
+	// but not if it is paused.
+	if c.State != "running" && c.State != "paused" {
+		// found container to reuse, but it is not in state running
+		if err := dc.Start(ctx); err != nil {
+			return dc, fmt.Errorf("start container %s in state %s: %w", req.Name, c.State, err)
+		}
+	}

OTOH, testcontainers-go does not provide native support for Pausing/Unpausing containers, unless using the underlying Docker client we expose (see my test).

Said that, I'm not sure how to proceed with this PR

[mdelapenya]

@georgespalding I added a commit with the related fix that I found

[stevenh]

A few clarifying questions