fix: do not limit pipeline_perf.py to nonci tests in metrics mode

When running A/B-Tests, the pipeline generated by pipeline_perf.py does
not care about pytest markers, it will run all tests specified and do
the A/B-comparision. However, when running the pipeline in metrics
collection mode, it passes `-m nonci` to pytest, which causes it to
filter out tests that do not have the `nonci` marker. Some of our
A/B-Tests (such as memory overhead) indeed do not have this marker,
leading to confusing errors (where everything works in A/B-mode, but if
just collecting metrics pytest fails with "no tests collected"). This
commit makes the behavior consistent, but also triggering non-nonci
tests in metrics collection mode.

Signed-off-by: Patrick Roy <[email protected]>

Author: roypat

.buildkite/pipeline_perf.py

@@ -73,7 +73,8 @@ def build_group(test):
         devtool_opts += " --ab"
         pytest_opts = f" {ab_opts} run {REVISION_A} {REVISION_B} --test {test_path}"
     else:
-        pytest_opts += f" -m nonci {test_path}"
+        # Passing `-m ''` below instructs pytest to collect tests regardless of their markers (e.g. it will collect both tests marked as nonci, and tests without any markers).
+        pytest_opts += f" -m '' {test_path}"
     binary_dir = test.pop("binary_dir")
     return group(
         label=test.pop("label"),