Timeout for test class that executes before/after on the same thread as test

[Nikita-Guchakov]

Hi!
We have flakily hanging tests, it happens rarely, mostly at CI, so it's hard to investigate which one test hangs.
I tried to deal with it by adding global timeouts, but it seems that it stuck uninterruptibly, SEPARATE_THREAD mode is not the solution, because some tests relies on thread locals, that were set up at lifecycle methods.

Another problem that developers may forget to clean threadlocals state (which maybe hidden deep in the business code) and then we have tests that broke each other.

It will be nice to have an option to run all test case methods with a single isolated thread, with timeout support as well.

Thanks

[marcphilipp]

It will be nice to have an option to run all test case methods with a single isolated thread, with timeout support as well.

If I understand you correctly, you're asking for something similar to #3939, is that correct?

[Nikita-Guchakov]

something similar

I don't get if this fully matched, I tried to read PR, but I don't know what is the scope of TestTask, is that all test methods of test class, or instance mode will be considered as well?

I will try to provide some pseudocode example :)

Assuming we have test class like this, with runs with junit.jupiter.execution.timeout.default=1m :

//at another test
val dirtyContext = ThreadLocal<String>().apply {
       it.set("value1")
}


class Test {
   val threadLocal2 = ThreadLocal<String>()

   @BeforeEach
   fun  setup() {
        threadLocal2.set("value2")
   }


   @Test
   fun test1() {
      while(dirtyContext.get() != null){
          // will stuck without SEPARATE_THREAD execution strategy
      }
      assertNonNull(threadLocal2.get()) // fails if execution strategy is SEPARATE_THREAD
   }

  @Test
  fun test2() {
      while(true){
         //stuck endlessly
      }
  }

}

Currently timeouts with SEPARATE_THREAD strategy will execute it like this:

main {
     val testInstance1 = Test()
     testInstance.setup()
     thread(timeout) {

     testInstance1.test1()
    }.join()
   

    val testInstance2 = Test()
    testInstance2.setup()
    thread(timeout) {
        testInstance.test2()
    }.join()
}

It will work with execution strategy that works like this:

main {
  thread(timeout) {
     val testInstance = Test()
     testInstance.setup()
     testInstance.test1()
  }.join()

  thread(timeout) {
     val testInstance = Test()
     testInstance.setup()
     testInstance.test2()
  }.join()
}

so

1. test1 will pass because of thread local properly isolated from other test, but not from setup methods,
2. test 2 will timed out and we be able to discover the issue

[marcphilipp]

Have you tried using @ResourceLock to avoid running tests using the same ThreadLocals in parallel?

[Nikita-Guchakov]

We don't run them in parallel, we serial execution. The question is not only about thread locals actually, but any tests that can stuck.
Some tests stuck randomly, our CI fail build by timeout and we don't know which test is flaky -> we want to be safe from hanging and be able to discover flakily hanging tests -> we enable global timeout -> that wont help, because test stucks uninteruptibely -> we enable SEPARATE_THREAD mode -> our test that relies on thread locals fails now

Even more, test may be both handing and relying on thread locals, which makes timeouts on separated thread unappliable for them at all

[marcphilipp]

Ok, I understand the problem now, thanks for the explanation! But a separate thread per test invocation (with same thread timeouts) would also not help if tests get stuck and don't respond to interrupts, would it?

[Nikita-Guchakov]

@marcphilipp it works currently for SEPARATE_THREAD

        @Test
	@Timeout(value = 200, unit = TimeUnit.MILLISECONDS)
	fun stucks() {
		while (true){}
	}

	@Test
	@Timeout(value = 200, unit = TimeUnit.MILLISECONDS, threadMode = Timeout.ThreadMode.SEPARATE_THREAD)
	fun fails() {
		while (true){}
	}

[Nikita-Guchakov]

What I'm expecting here is another mode, like

junit.jupiter.execution.timeout.default=1m
junit.jupiter.execution.threadMode.default=SEPARATED_THREAD_FOR_WHOLE_TEST_LYFECYCLE :)

which would work as I explained before, both spawning new thread, applying timeouts as for SEPARATE_THREAD, but calling lifecycle methods like before/after on the same thread as test

[Nikita-Guchakov]

Or maybe there can be a flag to abort (or continue on another thread) tests execution after waiting for interruption for a while.
If there is a thread that sends interruption already I believe it's doable without too much of rewriting :( Dumping stacktrace in that case will be helpful too. All of this will be enough to find out which test were stuck.
Don't know... The goal is to find out stuck tests same way as we find out failing tests, without breaking current tests that depends on thread locals