Provide a reliable way to report output from multithreaded and parallel tests

[joffrey-bion]

TL;DR: I propose to:

1. Add an API to report stdout and stderr from tests using the TestReporter, different from publishEntry("stdout", "..."). For example, provide PrintStreams like TestReporter.out and TestReporter.err so the test authors can do TestReporter.out.println("...") instead of System.out.println().
2. Add a callback to TestExecutionListener to get streaming output coming from these new streams (as proposed in Notify TestExecutionListener of test stdout/stderr output continuously instead of all at the end #4317), or maybe some other reporting keys (different from stdout/stderr) to get this output in reportingEntryPublished.

---

At the moment, JUnit doesn't recognize output from other threads than the original test thread, or at least it doesn't attribute them to the test it originates from. As we can see in the docs:

Please note that the captured output will only contain output emitted by the thread that was used to execute a container or test. Any output by other threads will be omitted because particularly when executing tests in parallel it would be impossible to attribute it to a specific test or container.

Properly dealing with output from System.out.println() on JUnit side would require something like Kotlin's CoroutineContext, and I'm not sure there is an equivalent in pure Java. So I get that it might be effectively impossible for JUnit to attribute regular output to a test (when using parallel execution).

Nevertheless, test authors need a reliable way to print things from multithreaded tests while also ensuring it will be attributed to the proper test in reports and in listeners if run in parallel. While using System.out and System.err is not possible, nothing prevents JUnit from providing a TestReporter.out / TestReporter.err mechanism, as mentioned in this comment: #780 (comment)

Alternatives that don't seem appropriate:

• TestReporter.publishEntry(String) just publishes regular metadata under the value key. Tools cannot really decide whether this should just be printed to the console, or added to the metadata of the test with other key-value pairs.
• TestReporter.publishEntry("stdout", "...") seems like the perfect candidate at first glance, because it's exactly what is used by the stream capture mechanism to report the captured output. One problem with this is that it's unclear whether the same key can be used repeatedly for each line of output (a key-value store probably accepts only one value per key?). More importantly, the StreamInterceptingTestExecutionListener reports the whole captured output in bulk at the end of the tests, which means that the output from background threads would come first, and then the output from the test thread will appear at the end, giving a very confusing ordering. See Notify TestExecutionListener of test stdout/stderr output continuously instead of all at the end #4317
• Also, more generally, the publish* APIs seem mostly meant for reports / report entries (semantically), and don't feel appropriate to stream output to test listeners

With a special way to print output via the TestReporter, the TestExecutionListener could have a new (pair of) callback(s) to receive stdout/stderr events that come either from the captured System.out/System.err from the main test thread, or from these new print streams. See #4317.
Alternatively, this could be achieved by piggy-backing on the existing reportingEntryPublished and using some new keys like stdout-stream and stderr-stream.

[joffrey-bion]

As a build tool author, I can provide any custom TestExecutionListener, but I don't know in which form I should expect output from from the user. There is no documentation about the "standard" keys of the report entries and their semantics, so I don't think I'm supposed to respond to value, stdout, or stderr keys in any special way.

[marcphilipp]

Nevertheless, test authors need a reliable way to print things from multithreaded tests while also ensuring it will be attributed to the proper test in reports and in listeners if run in parallel. While using System.out and System.err is not possible, nothing prevents JUnit from providing a TestReporter.out / TestReporter.err mechanism, as mentioned in this comment: #780 (comment)

I think that's an interesting idea!

We have the same problem in our integration tests which trigger third-party tools like Gradle and Maven. For now, we're attaching their output as files using the new file attachment feature that will be released in 5.12 (see OutputAttachingExtension).

As a build tool author, I can provide any custom TestExecutionListener, but I don't know in which form I should expect output from from the user. There is no documentation about the "standard" keys of the report entries and their semantics, so I don't think I'm supposed to respond to value, stdout, or stderr keys in any special way.

That's true and partially due to now build tool consuming them in a meaningful way so far. You could make the case that's because they're not documented so we should definitely do that.

There is now some special treatment in OpenTestReportGeneratingListener:

junit5/junit-platform-reporting/src/main/java/org/junit/platform/reporting/open/xml/OpenTestReportGeneratingListener.java

Lines 355 to 360 in ee52c64

	if (keyValuePairs.containsKey(STDOUT_REPORT_ENTRY_KEY)
	|| keyValuePairs.containsKey(STDERR_REPORT_ENTRY_KEY)) {
	attachOutput(attachments, entry.getTimestamp(), keyValuePairs.get(STDOUT_REPORT_ENTRY_KEY),
	"stdout");
	attachOutput(attachments, entry.getTimestamp(), keyValuePairs.get(STDERR_REPORT_ENTRY_KEY),
	"stderr");

I'll bring this up in the next team sync call.

[joffrey-bion]

Thanks a lot @marcphilipp for your quick reply and your willingness to discuss this.

You could make the case that's because they're not documented so we should definitely do that. There is now some special treatment in OpenTestReportGeneratingListener

Indeed I noticed the OpenTestReportGeneratingListener and its usage of the stdout / stderr keys. That's partially why I was tempted to use them to stream the output from the tests to my own listener. But, as I described, using this key for streaming output wouldn't work well in combination with the bulk-report-at-the-end behavior from StreamInterceptingTestExecutionListener. All user-reported output via testReporter.publishEntry("stdout", "my output line") would come before all the output printed via System.out, and the order is important in most cases, so that's not viable.

I disabled the built-in output capture (and thus the StreamInterceptingTestExecutionListener) since I implemented an alternative that actually streams the output in the same format my listener does, but I can't prevent users from enabling the output capture on their own (they might want it). So I still cannot expect users to use the same stdout/stderr keys to report their output from other threads, because that would mess up the ordering from the captured output as described above. Hence why I would need a different dedicated way to stream test output from the user's side.

I liked the TestReporter.out/err suggestion, but I'm definitely open to alternative solutions, like using other special keys like stdout-line and stderr-line, or stdout-stream and stderr-stream.