Users expect `[options.packages.find] exclude` to be recursive, but it is not (adding `.*` to the pattern is needed)

[KOLANICH]

setuptools version

57.0.1.dev3+ga5131f0b.post.20210528a

Python version

cpython 3.9

OS

Ubuntu 21.04

Additional environment information

No response

Description

[options.packages.find]
exclude =
    arpeggio.tests

doesn't work. In fact it doesn't work in a lot of other projects. But in other projects there usually no more than 2 levels in the excluded dir, so the workaround tests and tests.* works.

Expected behavior

Should exclude the whole hierarchy if a dir is mentioned.

How to Reproduce

https://github.com/textX/Arpeggio/tree/f24ceb0e79f6c9216dc035de123a388d113a332e

Output

....
adding 'arpeggio/tests/__init__.py'
adding 'arpeggio/tests/test_decorator_combine.py'
adding 'arpeggio/tests/test_default_semantic_action.py'
adding 'arpeggio/tests/test_eolterm.py'
adding 'arpeggio/tests/test_error_reporting.py'
adding 'arpeggio/tests/test_examples.py'
....

Code of Conduct

• I agree to follow the PSF Code of Conduct

[exhuma]

I'm having a very similar issue: When running pytproject-build, the resulting sdist has the tests folder included even though it is explicitly excluded using the exclude=tests,test* config. Interestingly, the .whl file does not include the tests.

As I understand, pyproject-build first creates an sdist, and then creates the wheel from that sdist. So the result surprises me.

Example contents of the sdists (notice that it includes the test-files):

vscode ➜ /workspaces/puresnmp (develop ✗) $ tar tzvf dist/puresnmp-1.10.2.post1.tar.gz 
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/
-rw-r--r-- vscode/vscode  1069 2021-07-11 11:28 puresnmp-1.10.2.post1/LICENSE.txt
-rw-r--r-- vscode/vscode    90 2021-07-11 11:28 puresnmp-1.10.2.post1/MANIFEST.in
-rw-r--r-- vscode/vscode  3240 2021-07-11 12:16 puresnmp-1.10.2.post1/PKG-INFO
-rw-r--r-- vscode/vscode  2280 2021-07-11 11:29 puresnmp-1.10.2.post1/README.rst
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/puresnmp/
-rw-r--r-- vscode/vscode   708 2021-07-11 11:39 puresnmp-1.10.2.post1/puresnmp/__init__.py
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/puresnmp/aio/
[...]
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/tests/
-rw-r--r-- vscode/vscode  7886 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/__init__.py
-rw-r--r-- vscode/vscode   442 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/asyncmock.py
-rw-r--r-- vscode/vscode   115 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/conftest.py
drwxr-xr-x vscode/vscode     0 2021-07-11 12:16 puresnmp-1.10.2.post1/tests/external_api/
-rw-r--r-- vscode/vscode     0 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/external_api/__init__.py
-rw-r--r-- vscode/vscode 16904 2021-07-11 11:29 puresnmp-1.10.2.post1/tests/external_api/test_v1.py
[...]

Contents of the wheel (no test files):

vscode ➜ /workspaces/puresnmp (develop ✗) $ unzip -l dist/puresnmp-1.10.2.post1-py3-none-any.whl 
Archive:  dist/puresnmp-1.10.2.post1-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      708  2021-07-11 11:39   puresnmp/__init__.py
      733  2021-07-11 11:29   puresnmp/const.py
     5438  2021-07-11 11:29   puresnmp/exc.py
    10450  2021-07-11 11:29   puresnmp/pdu.py
        0  2021-07-11 11:28   puresnmp/py.typed
     2353  2021-07-11 11:29   puresnmp/snmp.py
     5166  2021-07-11 11:29   puresnmp/transport.py
     5195  2021-07-11 11:29   puresnmp/types.py
      764  2021-07-11 11:29   puresnmp/typevars.py
     7046  2021-07-11 11:29   puresnmp/util.py
        0  2021-07-11 11:28   puresnmp/aio/__init__.py
     4461  2021-07-11 11:29   puresnmp/aio/transport.py
        0  2021-07-11 11:28   puresnmp/aio/api/__init__.py
     9106  2021-07-11 11:29   puresnmp/aio/api/pythonic.py
    22491  2021-07-11 11:29   puresnmp/aio/api/raw.py
        0  2021-07-11 11:28   puresnmp/api/__init__.py
    11618  2021-07-11 11:30   puresnmp/api/pythonic.py
    23267  2021-07-11 11:30   puresnmp/api/raw.py
     1069  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/LICENSE.txt
     3570  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/METADATA
       92  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/WHEEL
        9  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/top_level.txt
     1813  2021-07-11 12:16   puresnmp-1.10.2.post1.dist-info/RECORD
---------                     -------
   115349                     23 files

setup.cfg

vscode ➜ /workspaces/puresnmp (develop ✗) $ cat setup.cfg 
[metadata]
name = puresnmp
project_urls =
    Homepage = https://github.com/exhuma/puresnmp
; [...]

[options]
include_package_data = True
packages = find:
install_requires =
    t61codec >= 1.0, <2.0
    x690 == 0.2
    dataclasses; python_version < "3.7"

[options.packages.find]
exclude =
    tests
    tests.*

[abravalheri]

Hi @KOLANICH, thank you very much for bringing this topic for discussion.

The current implementation specifically suggests that this behaviour is intentional, although I don't know what is the rationale behind it. So I will re-label this from bug to enhancement.

(I can conjecture that if the exclusion is automatically recursive there is no way of "re-including" a nested package, so in some sense the current implementation is more general/flexible than the proposal - not sure how relevant this rationale is).

I don't know what would be the impacts of changing this behaviour (it would definitely not be backwards compatible), so I will leave the assessment of the proposal for the other maintainers.

[PabloEmidio]

@abravalheri nowadays and I am still facing this same issue.

project structure:

╰─ ls -R --ignore="venv" --ignore="__pycache__" --ignore="build" --ignore="dist" --ignore="*egg-info" 
.:
pyproject.toml  README.md  src  tests  VERSION

./src:
mproject

./src/mproject:
goal.py  __init__.py

./tests:
__init__.py  test_keep.py

pyproject.toml content:

[build-system]
requires = ["setuptools >= 67.2.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mproject"
authors = [{name = "Pablo Emidio S.S", email = "p.emidiodev@gmail.com"}]
description = "A test project structure"
readme = "README.md"
requires-python = ">=3.9"
keywords = ["One", "Two"]
license = {text = "MIT"}
dependencies = [
	'importlib-metadata; python_version<"3.10"'
]
dynamic = ["version"]

[tool.setuptools]
include-package-data = false

[tool.setuptools.packages.find]
where = ["src"]
include = ["mproject"]
exclude = ["tests", "tests.*"]

[tool.setuptools.dynamic]
version = {file = "VERSION"}

whel content:

╰─ unzip -l dist/mproject-0.0.1.dev0-py3-none-any.whl
Archive:  dist/mproject-0.0.1.dev0-py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      175  02-14-2023 00:59   mproject/__init__.py
       25  02-14-2023 01:00   mproject/goal.py
      320  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/METADATA
       92  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/WHEEL
        9  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/top_level.txt
      467  02-14-2023 01:39   mproject-0.0.1.dev0.dist-info/RECORD
---------                     -------
     1088                     6 files

tar content:

╰─ tar -tvf dist/mproject-0.0.1.dev0.tar.gz
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/
-rw-r--r-- pabloemidio/pabloemidio   260 2023-02-13 22:39 mproject-0.0.1.dev0/PKG-INFO
-rw-r--r-- pabloemidio/pabloemidio    17 2023-02-13 21:47 mproject-0.0.1.dev0/README.md
-rw-r--r-- pabloemidio/pabloemidio    10 2023-02-13 22:02 mproject-0.0.1.dev0/VERSION
-rw-r--r-- pabloemidio/pabloemidio   626 2023-02-13 22:28 mproject-0.0.1.dev0/pyproject.toml
-rw-r--r-- pabloemidio/pabloemidio    38 2023-02-13 22:39 mproject-0.0.1.dev0/setup.cfg
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/src/
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject/
-rw-r--r-- pabloemidio/pabloemidio   175 2023-02-13 21:59 mproject-0.0.1.dev0/src/mproject/__init__.py
-rw-r--r-- pabloemidio/pabloemidio    25 2023-02-13 22:00 mproject-0.0.1.dev0/src/mproject/goal.py
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/
-rw-r--r-- pabloemidio/pabloemidio   260 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/PKG-INFO
-rw-r--r-- pabloemidio/pabloemidio   276 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/SOURCES.txt
-rw-r--r-- pabloemidio/pabloemidio     1 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/dependency_links.txt
-rw-r--r-- pabloemidio/pabloemidio    47 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/requires.txt
-rw-r--r-- pabloemidio/pabloemidio     9 2023-02-13 22:39 mproject-0.0.1.dev0/src/mproject.egg-info/top_level.txt
drwxr-xr-x pabloemidio/pabloemidio     0 2023-02-13 22:39 mproject-0.0.1.dev0/tests/
-rw-r--r-- pabloemidio/pabloemidio    35 2023-02-13 22:02 mproject-0.0.1.dev0/tests/test_keep.py

---

package has being published in https://test.pypi.org/project/mproject/ to maybe help debbuging proposes.