Optimizing integration testing with ginkgo + testcontainers: reducing resource usage by sharing a single container across test suites

[violettran]

Hi, my team has been using Ginkgo v2 with Testcontainers for integration testing for almost a year. It's been very convenient and easy to write tests, but I noticed that as the number of tests increased, the number of containers also grew. Consequently, our AWS test servers saw doubled RAM and CPU usage.

To address this, I tried using a single container per test suite to reduce the number of containers. This worked smoothly for a while but eventually slowed down as new suites were added.

Now, my idea is to create a single container for all suites, with each suite running in a separate database within that container. However, your documentation mentions that suites are designed to run in isolation, and I couldn't find any guidance on determining when all test suites have finished running. How can I implement this?

[onsi]

hey @violettran

When running in parallel it is possible for any spec to be running at the same time as any other spec. Ginkgo simply requires that you write your tests in a way that handles this reality. How you do that is entirely up to you and will depend on the nature of the system under tests.

For the strictest isolation perhaps your test spins up a unique container and runs against it. I've seen some usecases where entire VMs or even k8s clusters are spun up for each test. It all depends on what's being tested.

I'm not actively working in the container/distributed systems space these days - so my integration tests tend to be application-level. I don't bother with container isolation because I don't need it. ginkgo's process-level isolation is more than enough. For cases when i'm using a database I simply spin up one shared database and generate shards within that database for each test process.

some patterns are discussed in more detail here

and I couldn't find any guidance on determining when all test suites have finished running

i'm not sure what you're referring to here and it sounds like you have a particular idea in mind. if you could share more detail about your ideas perhaps i can help. when you run, say, ginkgo -r -p it runs all your suites sequentially (but the specs in a given suite run in parallel with each other). "the test suites are all finished running" when ginkgo exits.