-
Notifications
You must be signed in to change notification settings - Fork 406
Introduce JUnit5 #2493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce JUnit5 #2493
Conversation
It seems that the current setup does not integrate with the jacocotestreport really well. |
Is junit-team/junit5#1024 related? |
@@ -11,6 +20,7 @@ plugins { | |||
} | |||
|
|||
apply from: 'gradle/scripts/yaml.gradle' | |||
apply plugin: 'org.junit.platform.gradle.plugin' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ugh, but at least the JUnit team is aware of the issue: junit-team/junit5#768. However, it appears to have been pushed back to 5.1 as of 14 hours ago. 😞
Note that there's a workaround listed in the issue, but I'm not sure if it's worth introducing that just to keep build.gradle
clean.
Nit: Pasting the buildscript
configuration from the JUnit docs introduced mixed tabs/spaces into this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tests do run on intellij FWIW, though had a result that was a good bit slower (~30s) than compared to master. The tests are overall too slow to effectively evaluate this changing, I think this is blocked until we get our tests running in a reasonable 15s ~ 20s ballpark. Then we can more easily see if there is a real performance difference. Adding 30s to a test run is a non-started, so I'd like to verify that result does not carry over once we fix the test execution speed.
It's really good to see your tenacity to get this working @RoiEXLab. It's valuable to get to a newer technology when we can. |
@ssoloff |
@ssoloff
I think this is kotlin code, any suggestions on how to apply it? |
Codecov Report
@@ Coverage Diff @@
## master #2493 +/- ##
============================================
+ Coverage 20.56% 20.56% +<.01%
- Complexity 5848 5860 +12
============================================
Files 831 831
Lines 73614 73618 +4
Branches 12400 12401 +1
============================================
+ Hits 15136 15141 +5
Misses 56389 56389
+ Partials 2089 2088 -1
Continue to review full report at Codecov.
|
@RoiEXLab I pushed a commit to your branch that implements the workaround in the issue you referenced. Some other things I noted while playing around here:
I suspect (2) and (3) probably just require changing the
But, unfortunately, I agree with @DanVanAtta that (1) is kind of a show-stopper until we can figure out how to improve the performance. Maybe it's just due to the fact that the vast majority of the tests have to run through the vintage test runner, so there's extra indirection there? Dunno... |
@ssoloff It's possible to update all tests at once to compile with junit 5 with some basic find and replace. (replacing imports is sufficient) |
According to the JUnit 5 Javadocs, the order is expected followed by actual, which is the same as in JUnit 4, and I believe has been that way since at least JUnit 3. Are you talking about something different? We may be using the API incorrectly (I've observed that in a few places), but it will be the same incorrect usage between JUnit 4 and 5. The only real problem is that if an assertion fails, the message printed will contain reversed values for expected and actual, and we can just fix those incrementally as we revisit tests. |
For issue (4), I pushed another commit that prevents the For issue (2), there appears to be no way to redirect stdout/stderr at this time (see junit-team/junit5#780). 😞 For issue (3), there is a known limitation that prevents the JUnit 5 runner's results from being included in the standard Gradle test report. 😞 |
@ssoloff Interesing I could swear they were reversed at some point. In this case I will upgrade them at once. |
Isn't there a hamcrest style API/library built into Junit5 that avoids the expected and actual ordering problem by being type safe? It is one of the perhaps really nice features of hamcrest that you do not have to worry about it so much. |
Running all tests (no integration) from IntelliJ, got the following results (all in seconds):
I switched over to Junit5 and did the same but only three times with the following results:
I remember when we had Junit5 previously for a bit that it was a bit slower, but not so drastically. Previously as well IntelliJ was running Junit5 by essentially running the build script and reporting back the results. Checking gradle builds as well on my system, the tests seem to still be running slow. I'm afraid this might be a nail in the coffin for the moment. |
There's no longer an assertThat method by JUnit 5. |
I updated all tests to JUnit5. @DanVanAtta I ran the tests, and on my machine they run equally fast using JUnit 5 and 4 (maximum 3 seconds difference), would be nice if you could verify the same. Now there are just 2 (maybe 3) issues left. For whatever reason javadoc errors for the integTests showed up, I have no idea why, shouldn't the test classes just get ignored? |
good work, can confirm performance is there again:
|
I'm guessing this is due to the JUnit 5 test task no longer being of type |
My experience seems to be different. Running on |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for slogging through the conversions, @RoiEXLab. It required more work than just a simple search-and-replace. 😄
public void setup() throws IOException { | ||
final File testFile = tempFolderRule.newFile(); | ||
final File testFile = tempFolderRule.newFile("testname"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Recommend changing the argument to getClass().getName()
.
public void setUp() throws Exception { | ||
file = temporaryFolder.newFile(); | ||
file = temporaryFolder.newFile("filename1123"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Recommend changing the argument to getClass().getName()
.
public void setUp() throws Exception { | ||
testObj = new FileSystemAccessStrategy(); | ||
final String text = DownloadFileProperties.VERSION_PROPERTY + " = 1.2"; | ||
mapFile = temporaryFolder.newFile(); | ||
mapFile = temporaryFolder.newFile("someothertestname"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Recommend changing the argument to getClass().getName()
.
@@ -105,7 +108,8 @@ public void shouldReturnFalseWhenObjectsAreNotEqual() { | |||
new FakeNestedClass(new FakeClass(-42)))); | |||
} | |||
|
|||
private static final class FakeNestedClass { | |||
@Nested | |||
private final class FakeNestedClass { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class does not contain tests, and thus does not need to be annotated with @Nested
.
@@ -159,7 +164,8 @@ public void shouldReturnFalseWhenOwnerObjectsAreNotEqual() { | |||
assertFalse(equalsPredicate.test(owner1, owner2)); | |||
} | |||
|
|||
private static final class FakeOwnerClass { | |||
@Nested | |||
private final class FakeOwnerClass { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class does not contain tests, and thus does not need to be annotated with @Nested
.
@@ -168,7 +174,8 @@ public void shouldReturnFalseWhenOwnerObjectsAreNotEqual() { | |||
} | |||
} | |||
|
|||
private static final class FakeOwneeClass { | |||
@Nested | |||
private final class FakeOwneeClass { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class does not contain tests, and thus does not need to be annotated with @Nested
.
@@ -179,7 +186,8 @@ public void shouldReturnFalseWhenOwnerObjectsAreNotEqual() { | |||
} | |||
} | |||
|
|||
private static final class FakeClass { | |||
@Nested | |||
private final class FakeClass { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class does not contain tests, and thus does not need to be annotated with @Nested
.
It also can remain static
because it's a child of the top-level class.
You need to update the JavadocMethod rule configuration to recognize the new JUnit 5 annotations. In config/checkstyle/checkstyle.xml, you need to change this line: <property name="allowedAnnotations" value="Override, Test, BeforeClass, AfterClass, Before, After"/> |
Found the differences: The following top-level tests in
They are not run in JUnit 4 because the top-level class is declared The remaining differences are due to an erroneous @RoiEXLab Please change diff --git a/src/test/java/games/strategy/triplea/delegate/DelegateTest.java b/src/test/java/games/strategy/triplea/delegate/DelegateTest.java
index e4d12c4..1a5378d 100644
--- a/src/test/java/games/strategy/triplea/delegate/DelegateTest.java
+++ b/src/test/java/games/strategy/triplea/delegate/DelegateTest.java
@@ -3,6 +3,7 @@ package games.strategy.triplea.delegate;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
+import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import games.strategy.engine.data.GameData;
@@ -78,7 +79,7 @@ public class DelegateTest {
protected UnitType carrier;
protected Resource pus;
- @Test
+ @BeforeEach
public void setUp() throws Exception {
gameData = TestMapGameData.DELEGATE_TEST.getGameData();
british = GameDataTestUtil.british(gameData); Thus +6 tests for JUnit 5 and +4 tests for JUnit 4 ends up being +2 tests for JUnit 5, which is 926 - 924. |
@ssoloff I looked into the integTest problem. diff --git a/build.gradle b/build.gradle
index cd9a86a..bad0c12 100644
--- a/build.gradle
+++ b/build.gradle
@@ -59,7 +59,7 @@ sourceSets {
java.srcDir 'src/integ_test/java'
resources.srcDir 'src/integ_test/resources'
- compileClasspath = sourceSets.main.output + sourceSets.test.output + configurations.testRuntime
+ compileClasspath = sourceSets.main.output + sourceSets.test.output + configurations.testRuntime + configurations.junitPlatform
runtimeClasspath = output + compileClasspath
}
}
@@ -111,19 +111,22 @@ dependencies {
testRuntime 'org.junit.jupiter:junit-jupiter-engine:5.0.1'
}
-task integTest(type: Test) {
+task integTest(type: junitPlatformTest.getClass()) {
group = LifecycleBasePlugin.VERIFICATION_GROUP
description = 'Runs the integration tests.'
- testClassesDirs = sourceSets.integTest.output.classesDirs
classpath = sourceSets.integTest.runtimeClasspath
- binResultsDir = file("$buildDir/integration_test_results/binary/integTest")
+ //binResultsDir = file("$buildDir/integration_test_results/binary/integTest")
- reports {
+ /*reports {
reports.html.destination = file("${reports.html.destination}/$name")
reports.junitXml.destination = file("${reports.junitXml.destination}/$name")
- }
+ }*/
+ outputs.dir file("$buildDir/integration_test_results/binary/integTest")
+ main = org.junit.platform.console.ConsoleLauncher.class.getName()
+
+ args = ['-d', sourceSets.integTest.java.srcDirs]
mustRunAfter tasks.test
} TL;DR There's no easy way to do it, but there is a way. See junit-team/junit5#874 for more information |
waitFor(this::serverNodeToDisconnect); | ||
@Test | ||
public void shouldBeAbleToChatAcrossMultipleNodes() { | ||
assertTimeout(Duration.ofSeconds(15), () -> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a assertTimeoutPreemptively method, which fails instantly after the timeout limit is reached.
This method always lets the test methods finish first. Should we use the other method?
The same for the other test cases where this method is used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just learned that this is the Junit4 behaviour... Changing that now
https://github.com/junit-team/junit4/wiki/Timeout-for-tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, It looks like the rule-of-thumb is to use assertTimeout()
for things like performance tests (e.g. this code that never blocks needs to run within a certain amount of time), while assertTimeoutPreemptively()
should be used for anything that can possibly block.
When using JUnit 5, we can drop the catch exception library as JUnit 5 has a nice exception API on it's own. NullPointerException e = assertThrows(NullPointerException.class, () -> Preconditions.checkNotNull(null));
assertEquals("something", e.getMessage());
//or something like that |
Yup, I believe you mentioned that a few months ago, so I've been looking forward to switching to JUnit 5. 😄 This PR is getting pretty big, though. It might be easier to just do that in a follow-up PR, if you'd like. |
build.gradle
Outdated
@@ -95,15 +105,12 @@ dependencies { | |||
testCompile 'eu.codearte.catch-exception:catch-exception:2.0.0-ALPHA-1' | |||
testCompile 'nl.jqno.equalsverifier:equalsverifier:2.3.3' | |||
testCompile 'org.hamcrest:java-hamcrest:2.0.0.0' | |||
testCompile 'org.mockito:mockito-core:2.10.0' | |||
testCompile 'junit:junit:4.12' | |||
testCompile 'org.mockito:mockito-core:2.8.47' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
BTW, was just curious what triggered the need to downgrade Mockito?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bad merge conflict resolution, will fix that ^^
public void setUp() throws Exception { | ||
super.setUp(); | ||
@BeforeEach | ||
public void setupPlaceDelegate() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does JUnit 5 make guarantees on the order @BeforeEach
methods will be called? That is, does it ensure that a @BeforeEach
method in the superclass will be called before a @BeforeEach
method in a subclass?
EDIT: Nevermind, answered my own question:
@BeforeEach
methods are inherited from superclasses as long as they are not overridden. Furthermore,@BeforeEach
methods from superclasses will be executed before@BeforeEach
methods in subclasses.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I should have provided a link to the JUnit docs
@ssoloff Of course, I planned to do that in a follow-up PR, just wanted to mention it. |
@ssoloff @DanVanAtta I figured out how to run our integrationTests with junit 5. This PR is now ready for a final review, I'd prefer minor issues to be adressed in a follo-up PR though, to avoid merge conflicts as the diff is pretty large |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a minor issue when running the integration tests. It is scanning the entire classpath instead of just the integTest
source set's classpath. There's really nothing we can do about that because we're running JUnit outside of Gradle at this point.
The problem is that it's picking up a unit test class that happens to have the suffix IntegrationTest
(ProxyableObjectInputOutputStreamIntegrationTest
), so that the integTest
task has seven more tests running using JUnit 5 (68) than with JUnit 4 (61). The test class in question has that particular suffix because it was testing the integration between ProxyableObjectOutputStream
and ObjectInputStream
, but it is still a "fast" test (because everything is done in memory), which is why it is included in the test
source set.
I think the easiest thing at this point is to just rename that test. For lack of a better idea, my suggestion would be to name it ProxyableObjectInputOutputTest
.
build.gradle
Outdated
@@ -106,24 +106,21 @@ dependencies { | |||
testCompile 'nl.jqno.equalsverifier:equalsverifier:2.3.3' | |||
testCompile 'org.hamcrest:java-hamcrest:2.0.0.0' | |||
testCompile 'org.mockito:mockito-core:2.11.0' | |||
testCompile 'org.junit.platform:junit-platform-launcher:1.0.1' | |||
testRuntime 'org.junit.platform:junit-platform-launcher:1.0.1' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Could you please move this down one line so we keep the different configurations grouped together? It's easy to miss the difference between testCompile
and testRuntime
when scanning the list. In fact, maybe an extra line of whitespace between them would be helpful, just as we did between compile
and testCompile
above.
build.gradle
Outdated
} | ||
main = 'org.junit.platform.console.ConsoleLauncher' | ||
args '--details', 'none' | ||
args "--reports-dir=$project.buildDir/integ-test-results/junit-platform" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: The project.
qualifier is superfluous here; $buildDir
should be sufficient and is consistent with the rest of the build script.
@RoiEXLab I missed your request to defer minor issues until a follow-up PR. I consider my suggestions to be minor. If you want to wait, that's fine with me. I'll change my review to Approved, but leave the PR open for @DanVanAtta. |
Given that @ssoloff already approved this PR, I'm going ahead and merge this in order to not block any further work. Please go ahead and reply with potential concerns on the follow-up PR.
This PR has been merged. |
Note that you can still run |
See #935 and #951 for more information.
This PR introduces the JUnit5 Runner now that eclipse officially supports JUnit5 in the latest 4.7.1a build.
This means you are highly encouraged to write new tests using the new namespace (importing the test annotation from junit5), and migrate the junit 4 test cases every now and then.
This is still experimental, it is working for me, but I really want to hear some feedback...