Failing test on CI on GitHub runners: `integration-tests/hibernate-search-orm-elasticsearch-outbox-polling`

[holly-cummins]

I'm seeing consistent failures on the CI runs in my fork (but not in the AWS runners we use for the main repo):

Failed to execute goal io.fabric8:docker-maven-plugin:0.46.0:stop (docker-start) on project quarkus-integration-test-hibernate-search-orm-elasticsearch-outbox-polling: At least one exception thrown during container removal.

I'm seeing the failure across a few different branches, so I don't think it's anything in my code. Instead, I think it's the difference between AWS and GH runners we've seen in a few other cases (for example, #48004).

Here's a few runs where I've seen it:

• https://github.com/holly-cummins/quarkus/actions/runs/15692150960
• https://github.com/holly-cummins/quarkus/actions/runs/15781078568

[quarkus-bot]

/cc @gsmet (elasticsearch,hibernate-search), @loicmathieu (elasticsearch), @marko-bekhta (elasticsearch,hibernate-search)

[marko-bekhta]

😔 I'm not sure ... but looking at the logs:

2025-06-16T22:03:00.2569245Z [INFO] --- compiler:3.14.0:testCompile (default-testCompile) @ quarkus-integration-test-hibernate-search-orm-elasticsearch-outbox-polling ---
2025-06-16T22:03:00.4304934Z [INFO] Loaded from the build cache
2025-06-16T22:03:00.4404808Z [INFO] 
2025-06-16T22:03:00.4405883Z [INFO] --- docker:0.46.0:stop (docker-start) @ quarkus-integration-test-hibernate-search-orm-elasticsearch-outbox-polling ---
2025-06-16T22:03:02.8483626Z [INFO] DOCKER> [docker.io/elastic/elasticsearch:9.0.2] "elasticsearch": Stop and removed container 14107ca586b2 after 0 ms
2025-06-16T22:03:02.8520869Z [ERROR] DOCKER> At least one exception thrown during container removal.

This container that fails to stop was started by quarkus-integration-test-hibernate-search-orm-elasticsearch as a devservice (see the hash 14107ca586b2):

2025-06-16T22:01:52.2992444Z 2025-06-16 22:01:52,224 INFO  [tc.doc.io/.0.2] (build-17) Container docker.io/elastic/elasticsearch:9.0.2 is starting: 14107ca586b2c5bcec84bead079c42b17839aadc9a9e9fc6bb6069c05e7805b5
2025-06-16T22:02:11.7484165Z 2025-06-16 22:02:11,661 INFO  [tc.doc.io/.0.2] (build-17) Container docker.io/elastic/elasticsearch:9.0.2 started in PT19.772695328S
2025-06-16T22:02:11.7486826Z 2025-06-16 22:02:11,662 INFO  [io.qua.ela.res.com.dep.DevServicesElasticsearchProcessor] (build-17) Dev Services for Elasticsearch started. Other Quarkus applications in dev mode will find the server automatically. For Quarkus applications in production mode, you can connect to this by configuring your application to use localhost:32852
2025-06-16T22:02:11.7510494Z 2025-06-16 22:02:11,664 INFO  [io.qua.hib.sea.orm.ela.dep.HibernateSearchElasticsearchProcessor] (build-28) Setting quarkus.hibernate-search-orm.schema-management.strategy=drop-and-create-and-drop to initialize Dev Services managed Elasticsearch server

So it seems that dev service started ES, then the docker plugin, which tries to stop and remove all elasticsearch containers (https://dmp.fabric8.io/#docker:stop)

allContainers --- Stops and removes any container that matches an image defined in the current project’s configuration.

tries to remove the one that dev service started, but it is already shutting down ?

what if we switch these search tests to also use dev services instead of the docker plugin, would that help?

[marko-bekhta]

Opened this draft, just in case: #48512