Test runner should allow specific subtest filtering, via --test-name-pattern or otherwise

[connor4312]

Version

19.3.0

Platform

Darwin MacBook-Pro-2.local 22.3.0 Darwin Kernel Version 22.3.0: Thu Jan 5 20:53:49 PST 2023; root:xnu-8792.81.2~2/RELEASE_X86_64 x86_64

Subsystem

test_runner

What steps will reproduce the bug?

Given a set of tests like this

describe("my suite 1", () => {
  it("works", async () => {
    // don't run me
  });

  describe("my suite 2", () => {
    it("works", async () => {
      // run me!
    });
  })
});

... it is impossible to command line filter (--test-name-filter) to run only the "works" test in my suite 2.

Being able to be precise about tests to run is important when working on an editor integration. If a user asks to "run this test", then only that test should run. This is especially relevant in the use case of integration tests, where running tests may have side effects.

How often does it reproduce? Is there a required condition?

100%

What is the expected behavior?

I should be able to run my suite 2 > works by itself. In every test runner I've worked with (jest, mocha, or vitest for example) names are generated for glob purposes by space-delimiting nested tests. So I expected to be able to pass --test-name-pattern="^my suite 1 my suite 2 works$" to run only that test.

(This does mean that identically named tests in the same context cannot be differentiated. But this is a common problem across test runners that also affects result output, and I already have code that emits a diagnostic warning ('yellow squiggle') if the user does this.)

What do you see instead?

I must pass --test-name-pattern="^my suite 1$" --test-name-pattern="^my suite 2$" --test-name-pattern="^works$", which runs the undesired test.

Additional information

Apologies for the late followup on #42984. I didn't get a chance to work on the vscode integration for this for a few months, and only noticed this issue later.

[cjihrig]

I think we should be able to support this without any additional overhead other than the case where --test-name-pattern is used. In this block we already compare the test name against all of the user provided patterns. We could also split the user provided patterns on some separator (> in the example above) and using the parent pointer, loop until we know if we have a fully qualified match or not.

[richiemccoll]

I'll take a look at this if nobody else has started on it.

[loynoir]

Hi guys, how about test on concated names?

var describeNames = ['level1', 'level2'];
var itName = 'should pass'
var testNamePattern = /level1>level2>should pass/

testNamePattern.test([...describeNames,itName].join('>'))
// true

I feel simpler and more reliable than inventing pattern syntax.

[MoLow]

I think describe should not be filtered, and we should only filter tests, @cjihrig WDYT?

[connor4312]

All other runners that come to mind do filter both on describe and test as part of the same sequence, as in the example in the issue report.

[cjihrig]

I think describe should not be filtered, and we should only filter tests, @cjihrig WDYT?

I would expect describes to be included.

[MoLow]

I have checked, and these are the behaviors of other test runners:

• jest --testNamePattern always runs describe regardless of the pattern.
• mocha --grep - runs all describe regardless of the pattern.
• playwright --grep - runs all describe regardless of the pattern.
• ava, tape, node-tap - no describe

All other runners that come to mind do filter both on describe and test as part of the same sequence, as in the example in the issue report.

this seems to be incorrect @connor4312

I would expect describes to be included.

@cjihrig do you still expect that, given this information?

[cjihrig]

That would still be my expectation, but I guess I would be wrong 😄. I'm fine with following the behavior of everyone else here.

[targos]

Jest certainly supports filtering by describe. It's the main way I work with it.

[connor4312]

Ah, sorry, I misunderstood. describes are run since the grep pattern is a pattern, not a prefix, so all tests have to get enumerated by running all describe blocks. So yes I think your intention is correct in saying "describe should not be filtered"

[MoLow]

Ok, so In regards to the comment made by @targos, I have tested this:

describe("yes", function() {
  it("no", () => {});
  it("yes", () => {});

  describe("maybe", function() {
    it("no", () => {});
    it("yes", () => {});
  });
});

describe("no", function() {
  it("no", () => {});
  it("yes", () => {});

  describe("maybe", function() {
    it("no", () => {});
    it("yes", () => {});
  });
});

• playwright test --grep "yes": all ran besides no -> no, no -> maybe -> no
• mocha --grep "yes": all ran besides no -> no, no -> maybe -> no
• jest --testNamePattern=yes: all ran besides no -> no, no -> maybe -> no

meaning describe can be used as a filter, but only to rule in its descendants - not to rule them out