test: create e2e environment; stop testing spacy in unit tests

[anakin87]

Related Issues

• alternative solution for Move Spacy NamedEntityExtractorBackend to core integrations #9187

Proposed Changes:

• create a e2e hatch environment, which inherits from test and installs the additional dependencies needed for e2e tests (Spacy in this case)
• remove Spacy from test environment

How did you test it?

CI: unit, integration and e2e test correctly run.

Notes for the reviewer

CI speed improvements in this case are difficult to measure precisely due to GitHub runner variability, and I want to avoid cherry-picking data.

I verified locally that test and e2e are separate environments, with test not installing Spacy (and not downloading the 400 Mb model).

Even if the improvements are small or difficult to measure, I would still incorporate this change. The dependency resolution for Spacy is not always trivial and has caused CI slowdowns in the past (#9159).

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 14400855624

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 10 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-0.08%) to 90.304%

---

Files with Coverage Reduction	New Missed Lines	%
core/pipeline/async_pipeline.py	3	68.46%
components/extractors/named_entity_extractor.py	7	64.52%

Totals
Change from base Build 14399545434:	-0.08%
Covered Lines:	10673
Relevant Lines:	11819

---

💛 - Coveralls

[anakin87]

test inherit from default but does not need its dependencies (reno, ruff, ...).
Using dependencies here then allows to use extra-dependencies in the e2e environment.

[julian-risch]

If we want to prevent tool.hatch.envs.test from inheritting from tool.hatch.envs.default and only install the project dependencies because it doesn't need its dependencies (reno, ruff, ...) then we could add here

[tool.hatch.envs.test]
template = "test"

or

[tool.hatch.envs.test]
template = "project"

I believe the former should work based on https://hatch.pypa.io/1.9/config/environment/overview/#self-referential-environments

[anakin87]

Ok I've now made test a self-referential environment:

• previously, it inherited from default and overrode dependencies
• now it does not inherit from default; specifies its dependencies

it has a similar effect, but maybe it is clearer

[anakin87]

"spacy-curated-transformers>=0.2,<=0.3",: this is no longer needed.

[julian-risch]

Looks quite good to me already. I have two comments I would like to briefly discuss. believe we can change something so that test and e2 don't install reno, ruff etc if they are really not needed.
And I have a question about files_yaml and how it combines multiple file paths.

[julian-risch]

If we want to prevent tool.hatch.envs.test from inheritting from tool.hatch.envs.default and only install the project dependencies because it doesn't need its dependencies (reno, ruff, ...) then we could add here

[tool.hatch.envs.test]
template = "test"

or

[tool.hatch.envs.test]
template = "project"

I believe the former should work based on https://hatch.pypa.io/1.9/config/environment/overview/#self-referential-environments

[julian-risch]

LGTM! 👍

[anakin87]

Unfortunately, using a self-referential environment has some consequences on DataDog tracing: https://github.com/deepset-ai/haystack/actions/runs/14399778757/job/40383320064

I tried to debug this for some time, but it's not easy...

Since we haven't changed anything on the Tracer, it's not a big deal (we are not introducing bugs).

For this reason, we decided to NOT use a self-referential environment (but get the same effect by overriding dependencies).