Capture stdout/stderr and include in test reports

[maxwindiff]

Overview

Currently the JUnit Platform and JUnit Jupiter do not support capturing stdout/stderr emitted during tests. This is quite useful for troubleshooting test failures on CI servers, esp. if the failures are not reproducible locally.

Background: http://stackoverflow.com/questions/43151448/how-to-capture-stdout-stderr-in-junit-5-gradle-test-report/43259526

[nipafx]

Thinking about this... where would the captured output go? Into the report?

[sbrannen]

I think we would have to make the captured output available in TestExecutionResult which could then be used in report generation.

In addition, we could consider making the same data available to extensions in JUnit Jupiter, much akin to the OutputCapture rule for JUnit 4 in Spring Boot.

[smoyer64]

I've got a couple of extensions for testing methods that interact with stdio that I'm going to move into junit-pioneer. For System.out and System.err, the test can capture the method's outputs and make assertions against that output. the interplay between the proposed feature (which is ostensibly writing log messages to one of these output streams and an application which is naturally writing to the "console" could be "very interesting".

[mfulton26]

If concurrent test execution becomes supported then stdout/stderr will have much less value. I like how TestReporter is independent of such but System.out cannot be so. It is almost as if what is really needed is something like a TestReporter.out. Users would have to redirect their logging, etc. to TestReporter.out instead of System.out (and same for a TestReporter.err).

[sbrannen]

FYI: as a proof of concept, I ported the JUnit 4 based OutputCapture rule from Spring Boot to JUnit Jupiter here:

https://github.com/sbrannen/junit5-demo/blob/master/src/test/java/extensions/CaptureSystemOutput.java

[childnode]

releated? #1166

[marcphilipp]

Currently in progress as part of #60.

[sormuras]

#60 was closed. What about this issue @marcphilipp?

[sbrannen]

Good question.

Technically speaking, the information is now available to report generators via org.junit.platform.launcher.TestExecutionListener.reportingEntryPublished(TestIdentifier, ReportEntry), but that doesn't mean the information actually gets included any in reports (other than those generated by the ConsoleLauncher).

But... developers can obviously generate a custom report by implementing their own TestExecutionListener.

So I'm guessing we could close this issue.

However, we still may wish to address the lacking ability to access stderr and stdout in Jupiter tests, potentially via an extension in JUnit Pioneer.

Thoughts?

[marcphilipp]

I think we should properly include it in the XML report, if capturing is enabled, before closing this issue.

[sbrannen]

What do you mean by "properly"? 😉

We already include all report entries in the system-out element via org.junit.platform.console.tasks.XmlReportWriter.writeReportEntriesToSystemOutElement(TestIdentifier, XMLStreamWriter).

[sbrannen]

Tentatively slated for 5.3 RC1 in light of #60.

[marcphilipp]

What do you mean by "properly"?

We could use a separate system-out element for the actual system output (the XSD seems to allow that, not sure which tools support it) and exclude the report entry from the other report entries. Moreover, there's a system-err element that we should use for stderr output.

[sbrannen]

SGTM. Thanks for expounding.

[JLLeitschuh]

@marcphilipp Can this be integrated into the official Gradle Plugin somehow? What is the overhead required to achieve this?

[marcphilipp]

The official Gradle plugin does currently not even support parallel test execution... I think we should fix that first. 😉