Multiple test "groupings" - outputs and reports

[bradleygore]

👋🏻 Hey there!

I'm using ginkgo 2.21.0 on a codebase that has thousands of tests across a few hundred suites and in our CI (Github Actions) we basically have it broken out into 2 groups:

1. Tests that can be parallelized
2. Tests that cannot

We target each group via label filters. The first group is the largest - as that group has no dependency on a shared stateful resource like database, etc... The second group is the smallest. We run ginkgo from the root, so that the label filters do the work - which has been fantastic! Execution looks like this:

// first group
ginkgo -r --github-output --label-filter="!(db_integration || 3rd_party_integration)" --skip-package=$SKIP_PKGS \
--silence-skips --randomize-suites --fail-fast --trace --no-color --procs=4 --compilers=4 \
--cover --covermode=count --coverprofile=$COV_FILE --output-dir=./tests --junit-report=$JUNIT_FILE > $TESTS_OUT 2>&1

// second group
ginkgo -r --github-output --label-filter="db_integration" --randomize-suites --fail-fast --trace --no-color --skip-package=$SKIP_PKGS \
--silence-skips --cover --covermode=count --coverprofile=$COV_FILE --output-dir=./tests --junit-report=$JUNIT_FILE ./ >> $TESTS_OUT 2>&1

I've a couple questions on ways to make the final outputs a bit more streamlined. I'm already using --github-output, which was a huge improvement, but looking to go a bit further if possible.

1. ❓ Is it possible to have a Reporter that omits all of the 0/# suites?
   • For example, when the second (smaller) grouping is executed, we get tons of lines like this:
     Printer Suite - 0/23 specs SUCCESS! 259.292µs PASS
2. ❓ Because I'm running both groups in parallel (as separate jobs in Github Action workflow) I manually combine the junit outputs at the end like this go run github.com/nezorflame/[email protected] -o ./tests/combined.junit.xml $(ls -d $PWD/tests/* | grep "\.junit\.xml"). This results in a bit of an odd report in that it shows far more skips (and maybe even runs) than actual. So the output looks like this: 22467 ran | 11123 passed | 11344 skipped | 0 failed Is there a better way to combine the junit output? I understand that it's not ginkgo that's doing the combining, but curious if there's some way to manipulate the output so that label-filtered skips aren't reported as skips or something.

Thanks in advance 🙏🏻

[onsi]

hey @bradleygore

The first thing I’d suggest is you can use the Serial decorator to mark the serial tests and then just have everything work as a single ginkgo -r invocation. You'll lose the ability to run the serial specs in parallel to the parallel specs via different GitHub actions - but if that's a trade off you're willing to make then that could work.

For your first question - there isn't a mechanism to suppress that output, no. If you'd like to work on a PR I can point you at how we could add such a mechanism.

For your second question - here too I don't have a great answer for you unfortunately. JUnit is "just XML" so you could write some code to merge the reports by hand - but that could get gnarly. It might be easier to manipulate Ginkgo's JSON output in this way and then pass the merged Report object into GenerateJUnitReport. The tricky bit will be figuring out how to pair up specs when performing the merge but I can help with that if you'd like to pursue that option.

But there isn't anything "out of the box" so to speak. Just the Serial decorator.

[bradleygore]

Thanks @onsi - appreciate the response as well as the Ginkgo and Gomega tools you have built!

We initially had both groups of tests running together and used the Serial decorator, but one group takes ~10 minutes and the other group ~8 minutes. Splitting them makes our CI execution time faster as max(groupA, groupB) vs groupA + groupB run time.

I like the idea of coming at the second item from the JSON output. I may do some digging into that in my next free cycles. 😄