Issue calling Setup/Cleanup in tests

[benmccann]

Normally you can do some setup and tear down before and after your tests:

testOptions in Test += Tests.Setup( loader => ... )
testOptions in Test += Tests.Cleanup( loader => ... )

Note: When forking, the ClassLoader containing the test classes cannot be provided because it is in another JVM. Only use the () => Unit variants in this case.

So when your tests are forked like Play's you have to do this:

testOptions in Test += Tests.Setup( () => println("Setup") )
testOptions in Test += Tests.Cleanup( () => println("Cleanup") )

I want to start a DB process before my tests run and stop the process after. The issue I'm having is that I don't know how to share a variable between setup and cleanup so that I can take the DB that I started in setup and call the stop method on it in cleanup.

A suggestion would be to have Tests.Setup and Tests.Cleanup replaced by a Tests.Hook which takes a class providing Setup and Cleanup methods. This class could then have member variables shared between the two functions

[eed3si9n]

How would Tests.Hook look like?

testOption in Test += Test.Hook(
  new Hook {
    def setup: Unit = ()
    def cleanup: Unit = ()
  }
)

cleanup would still not know what database setup has set up.

[benmccann]

Wouldn't it? I thought it'd allow you to have a class variable that could be shared between the two methods like:

testOption in Test += Test.Hook(
  new Hook {
    db: DB = Option.None
    def setup = {
      db = DBManager.start();
    }
    def cleanup = {
      DBManager.teardown(db);
    }
  }
)

[eed3si9n]

With that, the single Hook object is shared across multiple tests right?
I am assuming you want multiple of hooks per fork, not just a single global variable.

[benmccann]

I want one Hook object shared across tests. I can do setup/cleanup on a per-test basis in Junit. But for really expensive stuff like start a DB process and stop it, I want to do it only once for the whole test suite and that can only be done in SBT. I believe this is how the Tests.Setup and Test.Cleanup already work today, so there'd be no behavior change there.

[eed3si9n]

I want to do it only once for the whole test suite and that can only be done in SBT. I believe this is how the Tests.Setup and Test.Cleanup already work today, so there'd be no behavior change there.

For Tests.Setup, there's a single instance of it that gets called each time. So the Tests.Setup object itself it stateless. Whereas in your example, I see that you're mutating the Hook object:

  def setup = {
    db = DBManager.start();
  }

I don't think that could work. Maybe:

testOption in Test += Test.Hook(
  new Hook {
    def setup(x: TestGroupFingerprintSomething): Unit = ()
    def cleanup(x: TestGroupFingerprintSomething): Unit = ()
  }
)

[benmccann]

For Tests.Setup, there's a single instance of it that gets called each time.

Are you saying it gets called each time the tests are run? Or is there some other operation that happens multiple times in a single test run?

[eed3si9n]

I think we were using the word "test" in two different ways. Let's avoid this term, and use one of the four:

1. test task on sbt
2. test grouping
3. test suite
4. test example, the individual function that asserts something

I meant that a single Hook object is shared across multiple runs of the test task.
In most situations a single run of the test task would correspond with a default test grouping containing all test suites. However, you could fork the test, assign different groupings for each test suites, and they will run concurrently, each calling the Setup.

IO> set fork in test := true
IO> set testOptions in Test += Tests.Setup( () => println("Setup") )
IO> set concurrentRestrictions in Global := Seq(Tags.limit(Tags.ForkedTestGroup, 4))
IO> set testGrouping in Test := ((definedTests in Test).value groupBy (_.name(4)) map { case (letter, tests) => new Tests.Group(letter.toString, tests, Tests.SubProcess(Seq("-Dfirst.letter"+letter)))} toSeq)
[info] Defining ioProj/test:testGrouping
[info] The new value will be used by ioProj/test:executeTests, ioProj/test:testOnly, ioProj/test:testQuick
[info] Reapplying settings...
[info] Set current project to IO (in build file:/Users/eugene/work/sbt/)
IO> show test:testGrouping
[info] List(Group(R,List(Test sbt.RichURISpecification : subclass(true, org.scalacheck.Properties), Test sbt.RichURISpecification : subclass(true, org.scalacheck.Prop)),SubProcess(ForkOptions(None,None,List(),None,List(-Dfirst.letterR),false,Map()))), Group(C,List(Test sbt.CheckStash : subclass(true, org.specs2.specification.SpecificationStructure), Test sbt.CopySpec : subclass(true, org.scalacheck.Properties), Test sbt.CopySpec : subclass(true, org.scalacheck.Prop)),SubProcess(ForkOptions(None,None,List(),None,List(-Dfirst.letterC),false,Map()))), Group(N,List(Test sbt.NameFilterSpecification : subclass(true, org.scalacheck.Properties), Test sbt.NameFilterSpecification : subclass(true, org.scalacheck.Prop)),SubProcess(ForkOptions(None,None,List(),None,List(-Dfirst.letterN),false,Map()))), Group(W,List(Test sbt.WriteContentSpecification : subclass(true, org.scalacheck.Properties), Test sbt.WriteContentSpecification : subclass(true, org.scalacheck.Prop)),SubProcess(ForkOptions(None,None,List(),None,List(-Dfirst.letterW),false,Map()))))
[success] Total time: 0 s, completed Jun 24, 2015 12:53:42 AM
IO> test
Setup
Setup
Setup
Setup

[benmccann]

Oh, I see. I didn't realize test groups were a thing that existed. What do people create groups for or when would you use one?

It seems like it could be helpful to have both a SetupGroup/CleanupGroup that gets called for each test group as well as a SetupRun/CleanupRun that gets called once per task. The thing that'd be most helpful to be able to do for my use case is start a DB process before any tests are run and stop it after all the tests are run. Regardless of how tests are grouped, I don't want to do that more than necessary since it's quite expensive.

Then I could use setupRun()/cleanupRun() to setup my database the way I was trying to in the previous example I gave without screwing things up in the case that there are test groups.

[eed3si9n]

Test group (I think) is for assigning test suits to different forked JVMs.

Ultimately if all you want is some sort of a global variable, why not make some in project/Util.scala or something?:

object Foo {
  var db: Option[String] = None
}

Then you can access it from what we already have:

testOptions in Test += Tests.Setup( () => Foo.db = Some("x") )
testOptions in Test += Tests.Cleanup( () => deleteDatabase(Foo.db.get) )

[benmccann]

Ok, I didn't realize you could do that. The problem I'm having with this suggestion though is that Setup and Cleanup are called in unpredictable orders. They're also called multiple times and I'd prefer to call them only once given that they're expensive.

You can see this by setting up a project with sub-projects (e.g. activator new play-multidomain-seed) and adding to settings in project/Common.scala:

    testOptions in Test += Tests.Setup( () => System.out.println("Setting up") ),
    testOptions in Test += Tests.Cleanup( () => System.out.println("Tearing down") )

First time I ran this:

Setting up
Setting up
Tearing down
Tearing down
Setting up
Setting up
Tearing down
Tearing down

Second time I ran this:

Setting up
Setting up
Setting up
Setting up
Tearing down
Tearing down
Tearing down
Tearing down

It'd be really valuable to have a version of Setup and Cleanup that's called only once each time test is run.

[eed3si9n]

You can see this by setting up a project with sub-projects (e.g. activator new play-multidomain-seed) and adding to settings in project/Common.scala...

ok. So you're more concerned about aggregated test tasks across the subprojects, rather than forking.
I don't know if we currently have a concept like "run this task X prior to delegating test task across any subproject" besides using commands, or sequential tasks.

[benmccann]

Hmm sequential tasks don't seem to do anything. I set testSetupTeardownTask := Def.sequential(testSetupTask, test in Test, testTeardownTask).value, but when I run sbt test the setup and teardown tasks aren't run. Maybe it just adds orderings, but not dependencies?