Introduce support for parallel test execution

This commit adds opt-in support for parallel test execution and
capturing output to `System.out` and `System.err`. Both features are
disabled by default but can be enabled and configured using
configuration parameters.

The implementation is based on the Fork/Join Framework and designed to
be reusable by other test engines that extend HierarchicalTestEngine.

The Jupiter API provides annotations to declare which shared resources a
test needs to access and in which way. Moreover, the execution mode of a
test can be influenced.

In addition, a number of TestExecutionListeners have been made
thread-safe.

The documentation subproject is now configured to execute tests in
parallel. All other subprojects will have to wait as Gradle currently
blows up when used with parallel test execution.

Co-authored-by: Leonard Brünings <[email protected]>
Co-authored-by: Christian Stein <[email protected]>

Resolves #60. Closes #1461.

Author: marcphilipp

documentation/src/docs/asciidoc/link-attributes.adoc

@@ -21,6 +21,8 @@ endif::[]
 :ConsoleLauncher:                   {javadoc-root}/org/junit/platform/console/ConsoleLauncher.html[ConsoleLauncher]
 //
 :DiscoverySelectors_selectMethod:   {javadoc-root}/org/junit/platform/engine/discovery/DiscoverySelectors.html#selectMethod-java.lang.String-[selectMethod(String) in DiscoverySelectors]
+:HierarchicalTestEngine:            {javadoc-root}/org/junit/platform/engine/support/hierarchical/HierarchicalTestEngine.html[HierarchicalTestEngine]
+:ParallelExecutionConfigurationStrategy: {javadoc-root}/org/junit/platform/engine/support/hierarchical/ParallelExecutionConfigurationStrategy.html[ParallelExecutionConfigurationStrategy]
 :TestEngine:                        {javadoc-root}/org/junit/platform/engine/TestEngine.html[TestEngine]
 //
 :Launcher:                          {javadoc-root}/org/junit/platform/launcher/Launcher.html[Launcher]
@@ -54,6 +56,7 @@ endif::[]
 :EnabledIfSystemProperty:           {javadoc-root}/org/junit/jupiter/api/condition/EnabledIfSystemProperty.html[@EnabledIfSystemProperty]
 :EnabledOnJre:                      {javadoc-root}/org/junit/jupiter/api/condition/EnabledOnJre.html[@EnabledOnJre]
 :EnabledOnOs:                       {javadoc-root}/org/junit/jupiter/api/condition/EnabledOnOs.html[@EnabledOnOs]
+:Execution:                         {javadoc-root}/org/junit/jupiter/api/parallel/Execution.html[@Execution]
 :ExecutionCondition:                {javadoc-root}/org/junit/jupiter/api/extension/ExecutionCondition.html[ExecutionCondition]
 :ExtendWith:                        {javadoc-root}/org/junit/jupiter/api/extension/ExtendWith.html[@ExtendWith]
 :ExtensionContext:                  {javadoc-root}/org/junit/jupiter/api/extension/ExtensionContext.html[ExtensionContext]
@@ -70,6 +73,7 @@ endif::[]
 :TestTemplate:                      {javadoc-root}/org/junit/jupiter/api/TestTemplate.html[@TestTemplate]
 :TestTemplateInvocationContext:     {javadoc-root}/org/junit/jupiter/api/extension/TestTemplateInvocationContext.html[TestTemplateInvocationContext]
 :TestTemplateInvocationContextProvider: {javadoc-root}/org/junit/jupiter/api/extension/TestTemplateInvocationContextProvider.html[TestTemplateInvocationContextProvider]
+:ResourceLock:                      {javadoc-root}/org/junit/jupiter/api/parallel/ResourceLock.html[@ResourceLock]
 //
 :DisabledCondition:                 {current-branch}/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/extension/DisabledCondition.java[DisabledCondition]
 :RepetitionInfoParameterResolver:   {current-branch}/junit-jupiter-engine/src/main/java/org/junit/jupiter/engine/extension/RepetitionInfoParameterResolver.java[RepetitionInfoParameterResolver]

documentation/src/docs/asciidoc/release-notes/release-notes-5.3.0-M1.adoc

@@ -3,7 +3,8 @@
 
 *Date of Release:* ❓
 
-*Scope:* ❓
+*Scope:* Parallel test execution, output capturing, test sources for dynamic tests as well
+as various minor improvements and bug fixes.
 
 For a complete list of all _closed_ issues and pull requests for this release, consult the
 link:{junit5-repo}+/milestone/23?closed=1+[5.3 M1] milestone page in the JUnit repository
@@ -34,6 +35,19 @@ on GitHub.
 
 ==== New Features and Improvements
 
+* Experimental support for capturing output printed to `System.out` and
+  `System.err` during test execution. This feature is disabled by default and can be
+  enabled using a configuration parameter (cf.
+  <<../user-guide/index.adoc#running-tests-capturing-output, User Guide>>).
+* Reusable support for parallel test execution for test engines that extend
+  `HierarchicalTestEngine`.
+  - `HierarchicalTestEngine` implementations may now specify a
+    `HierarchicalTestExecutorService`
+  - By default, a `SameThreadHierarchicalTestExecutorService` is used.
+  - Test engines may use `ForkJoinPoolHierarchicalTestExecutorService` to support
+    parallel test execution based on the Fork/Join Framework.
+  - `Node` implementations may provide a set of `ExclusiveResources` and an
+    `ExecutionMode` to be used by `ForkJoinPoolHierarchicalTestExecutorService`.
 * New overloaded variant of `isAnnotated()` in `AnnotationSupport` that accepts
   `Optional<? extends AnnotatedElement>` instead of `AnnotatedElement`.
 * New `--fail-if-no-tests` command-line option for the `ConsoleLauncher`.
@@ -64,6 +78,10 @@ on GitHub.
 
 ==== New Features and Improvements
 
+* Experimental support for parallel test execution. By default, tests are still executed
+  sequentially; parallelism can be enabled using a configuration parameter (please refer
+  to the <<../user-guide/index.adoc#writing-tests-parallel-execution, User Guide>> for
+  examples and configuration options).
 * New support for the IBM AIX operating system in `@EnabledOnOs` and `@DisabledOnOs`.
 * New `assertThrows` methods in `Assertions` provide a more specific failure message if
   the supplied lambda expression or method reference returns a result instead of throwing

documentation/src/docs/asciidoc/user-guide/launcher-api.adoc

@@ -78,6 +78,11 @@ is currently supported via Java's `java.util.ServiceLoader` mechanism. For examp
 in a file named `org.junit.platform.engine.TestEngine` within the `/META-INF/services` in
 the `junit-jupiter-engine` JAR.
 
+NOTE: `{HierarchicalTestEngine}` is a convenient abstract base implementation (used by
+the `{junit-jupiter-engine}`) that only requires implementors to provide the logic for
+test discovery. It implements execution of `TestDescriptors` that implement the `Node`
+interface, including support for parallel execution.
+
 [[launcher-api-listeners-custom]]
 ==== Plugging in Your Own Test Execution Listeners
 

documentation/src/docs/asciidoc/user-guide/running-tests.adoc

@@ -762,3 +762,28 @@ useful.
 | +(micro \| integration) & (foo \| baz)+
 | all _micro_ or _integration_ tests for *foo* or *baz*
 |===
+
+[[running-tests-capturing-output]]
+=== Capturing Standard Output/Error
+
+Since version 1.3, the JUnit Platform provides opt-in support for capturing output
+printed to `System.out` and `System.err`. To enable it, simply set the
+`junit.platform.output.capture.stdout` and/or `junit.platform.output.capture.stderr`
+<<running-tests-config-params, configuration parameter>> to `true`. In addition, you may
+configure the maximum number of buffered bytes to be used per executed test or container
+using `junit.platform.output.capture.maxBuffer`.
+
+If enabled, the JUnit Platform captures the corresponding output and publishes it as a
+report entry using the `stdout` or `stderr` keys to all registered
+`{TestExecutionListener}` instances immediately before reporting the test or container as
+finished.
+
+Please note that the captured output will only contain output emitted by the thread that
+was used to execute a container or test. Any output by other threads will be omitted
+because particularly when
+<<writing-tests-parallel-execution, executing tests in parallel>> it would be impossible
+to attribute it to a specific test or container.
+
+WARNING: Capturing output is currently an _experimental_ feature. You're invited to give
+it a try and provide feedback to the JUnit team so they can improve and eventually
+<<api-evolution, promote>> this feature.

documentation/src/docs/asciidoc/user-guide/writing-tests.adoc

@@ -1290,3 +1290,90 @@ The last method generates a nested hierarchy of dynamic tests utilizing
 ----
 include::{testDir}/example/DynamicTestsDemo.java[tags=user_guide]
 ----
+
+
+[[writing-tests-parallel-execution]]
+=== Parallel Execution
+
+By default, JUnit Jupiter tests are run sequentially in a single thread. Running tests in
+parallel, e.g. to speed up execution, is available as an opt-in feature since version 5.3.
+To enable parallel execution, simply set the set the
+`junit.jupiter.execution.parallel.enabled` configuration parameter to `true`, e.g. in `junit-platform.properties` (see <<running-tests-config-params>> for other options).
+
+Once enabled, the JUnit Jupiter engine will execute tests on all levels of the test tree
+fully in parallel according to the provided
+<<writing-tests-parallel-execution-config, configuration>> while observing the declarative
+<<writing-tests-parallel-execution-synchronization, synchronization>> mechanisms. Please
+note that the <<running-tests-capturing-output>> feature needs to enabled separately.
+
+WARNING: Parallel test execution is currently an _experimental_ feature. You're invited
+to give it a try and provide feedback to the JUnit team so they can improve and
+eventually <<api-evolution, promote>> this feature.
+
+[[writing-tests-parallel-execution-config]]
+==== Configuration
+
+Properties like the desired parallelism and the maximum pool size can be configured using
+a `{ParallelExecutionConfigurationStrategy}`. The JUnit Platform provides two
+implementations out of the box: `dynamic` and `fixed`. Alternatively, you may implement a
+`custom` strategy.
+
+To select a strategy, simply set the `junit.jupiter.execution.parallel.config.strategy`
+configuration parameter to one of the following options:
+
+`dynamic`::
+  Computes the desired parallelism based on the number of available processors/cores
+  multiplied by the `junit.jupiter.execution.parallel.config.dynamic.factor`
+  configuration parameter (defaults to `1`).
+
+`fixed`::
+  Uses the mandatory `junit.jupiter.execution.parallel.config.fixed.parallelism`
+  configuration parameter as desired parallelism.
+
+`custom`::
+  Allows to specify a custom `{ParallelExecutionConfigurationStrategy}` implementation via
+  the mandatory `junit.jupiter.execution.parallel.config.custom.class` configuration
+  parameter to determine the desired configuration.
+
+If no configuration strategy is set, JUnit Jupiter uses the `dynamic` configuration
+strategy with a factor of 1, i.e. the desired parallelism will equal the number of
+available processors/cores.
+
+[[writing-tests-parallel-execution-synchronization]]
+==== Synchronization
+
+In the `org.junit.jupiter.api.parallel` package, JUnit Jupiter provides two
+annotation-based declarative mechanisms to change the execution mode and allow for
+synchronization when using shared resources in different tests.
+
+If parallel execution is enabled, all classes and methods are executed concurrently by
+default. You can change the execution mode for the annotated element and its subelements
+(if any) by using the `{Execution}` annotation. The following two modes are available:
+
+`SAME_THREAD`::
+  Force execution in the same thread used by the parent. For example, when used on a test
+  method, the test method will be executed in the same thread as any `@BeforeAll` or
+  `@AfterAll` methods of the containing test class.
+
+`CONCURRENT`::
+  Execute concurrently unless a resource constraint forces execution in the same thread.
+
+In addition, the `{ResourceLock}` annotation allows to declare that a test class or
+method uses a specific shared resource that requires synchronized access to ensure
+reliable test execution.
+
+If the tests in the following example were run in parallel they would be flaky, i.e.
+sometimes pass and other times fail, because of the inherent race condition of
+writing and then reading the same system property.
+
+[source,java]
+----
+include::{testDir}/example/SharedResourcesDemo.java[tags=user_guide]
+----
+
+When access to shared resources is declared using this annotation, the JUnit Jupiter
+engine uses this information to ensure that no conflicting tests are run in parallel.
+
+In addition to the string that uniquely identifies the used resource, you may specify an
+access mode. Two tests that require `READ` access to a resource may run in parallel with
+each other but not while any other test that requires `READ_WRITE` access is running.

documentation/src/test/java/example/SharedResourcesDemo.java

@@ -0,0 +1,65 @@
+/*
+ * Copyright 2015-2018 the original author or authors.
+ *
+ * All rights reserved. This program and the accompanying materials are
+ * made available under the terms of the Eclipse Public License v2.0 which
+ * accompanies this distribution and is available at
+ *
+ * http://www.eclipse.org/legal/epl-v20.html
+ */
+
+package example;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.parallel.ExecutionMode.CONCURRENT;
+import static org.junit.jupiter.api.parallel.ResourceAccessMode.READ;
+import static org.junit.jupiter.api.parallel.ResourceAccessMode.READ_WRITE;
+import static org.junit.jupiter.api.parallel.Resources.SYSTEM_PROPERTIES;
+
+import java.util.Properties;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.parallel.Execution;
+import org.junit.jupiter.api.parallel.ResourceLock;
+
+// tag::user_guide[]
+@Execution(CONCURRENT)
+class SharedResourcesDemo {
+
+	private Properties backup;
+
+	@BeforeEach
+	void backup() {
+		backup = new Properties();
+		backup.putAll(System.getProperties());
+	}
+
+	@AfterEach
+	void restore() {
+		System.setProperties(backup);
+	}
+
+	@Test
+	@ResourceLock(value = SYSTEM_PROPERTIES, mode = READ)
+	void customPropertyIsNotSetByDefault() {
+		assertNull(System.getProperty("my.prop"));
+	}
+
+	@Test
+	@ResourceLock(value = SYSTEM_PROPERTIES, mode = READ_WRITE)
+	void canSetCustomPropertyToFoo() {
+		System.setProperty("my.prop", "foo");
+		assertEquals("foo", System.getProperty("my.prop"));
+	}
+
+	@Test
+	@ResourceLock(value = SYSTEM_PROPERTIES, mode = READ_WRITE)
+	void canSetCustomPropertyToBar() {
+		System.setProperty("my.prop", "bar");
+		assertEquals("bar", System.getProperty("my.prop"));
+	}
+}
+// end::user_guide[]

documentation/src/test/java/example/SlowTests.java

@@ -0,0 +1,126 @@
+/*
+ * Copyright 2015-2018 the original author or authors.
+ *
+ * All rights reserved. This program and the accompanying materials are
+ * made available under the terms of the Eclipse Public License v2.0 which
+ * accompanies this distribution and is available at
+ *
+ * http://www.eclipse.org/legal/epl-v20.html
+ */
+
+package example;
+
+// tag::user_guide[]
+import static org.junit.jupiter.api.parallel.ExecutionMode.SAME_THREAD;
+
+import java.util.stream.IntStream;
+
+import org.junit.jupiter.api.Disabled;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.parallel.Execution;
+
+@Disabled
+class SlowTests {
+
+	@Execution(SAME_THREAD)
+	@Test
+	void a() {
+		foo();
+	}
+
+	@Test
+	void b() {
+		foo();
+	}
+
+	@Test
+	void c() {
+		foo();
+	}
+
+	@Test
+	void d() {
+		foo();
+	}
+
+	@Test
+	void e() {
+		foo();
+	}
+
+	@Test
+	void f() {
+		foo();
+	}
+
+	@Test
+	void g() {
+		foo();
+	}
+
+	@Test
+	void h() {
+		foo();
+	}
+
+	@Test
+	void i() {
+		foo();
+	}
+
+	@Test
+	void j() {
+		foo();
+	}
+
+	@Test
+	void k() {
+		foo();
+	}
+
+	@Test
+	void l() {
+		foo();
+	}
+
+	@Test
+	void m() {
+		foo();
+	}
+
+	@Test
+	void n() {
+		foo();
+	}
+
+	@Test
+	void o() {
+		foo();
+	}
+
+	@Test
+	void p() {
+		foo();
+	}
+
+	@Execution(SAME_THREAD)
+	@Test
+	void q() {
+		foo();
+	}
+
+	@Test
+	void r() {
+		foo();
+	}
+
+	@Test
+	void s() {
+		foo();
+	}
+
+	private void foo() {
+		IntStream.range(1, 100_000_000).mapToDouble(i -> Math.pow(i, i)).map(Math::sqrt).max();
+	}
+}
+// end::user_guide[]

documentation/src/test/resources/junit-platform.properties

@@ -0,0 +1,3 @@
+junit.jupiter.execution.parallel.enabled=true
+junit.jupiter.execution.parallel.config.strategy=fixed
+junit.jupiter.execution.parallel.config.fixed.parallelism=6