Transitive constraints are not properly considered in multiple output recipes

[jjhelmus]

Transitive package constraints from a sub-package do not appear to be considered when rendering recipes with multiple outputs. This can result in an unnecessary UnsatisfiableError during the build.

For example for a recipe with two outputs:

package:
  name: foo_split
  version: 1.0.0

outputs:
  - name: libfoo
    requirements:
      host:
        - openssl

  - name: foo
    requirements:
      host:
        - python
        - {{ pin_subpackage('libfoo', exact=True) }}

Attempting to pin openssl to anything but the latest version will result in foo failing to build. For example with the following CBC.yaml file:

openssl:
  - 1.0.2
python:
  - 3.6

The recipe renders as:

$ conda render .
read filter "zstd" is not supported
write filter "zstd" is not supported
--------------
Hash contents:
--------------
{'openssl': '1.0.2'}
----------
meta.yaml:
----------
package:
    name: libfoo
    version: 1.0.0
build:
    noarch: false
requirements:
    host:
        - ca-certificates 2018.03.07 0
        - libgcc-ng 8.2.0 hdf63c60_1
        - openssl 1.0.2p h14c3975_0
    run:
        - openssl >=1.0.2p,<1.0.3a
outputs:
    -   name: libfoo
        requirements:
            host:
                - openssl
    -   name: foo
        requirements:
            host:
                - python
                - libfoo 1.0.0 0
extra:
    copy_test_source_files: true
    final: true
    parent_recipe:
        name: foo_split
        path: /home/jhelmus/workspace/one_offs/cb_multi_out_bug
        version: 1.0.0

--------------
Hash contents:
--------------
{}
----------
meta.yaml:
----------
package:
    name: foo
    version: 1.0.0
build:
    noarch: false
requirements:
    host:
        - ca-certificates 2018.03.07 0
        - libgcc-ng 8.2.0 hdf63c60_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - libffi 3.2.1 hd88cf55_4
        - ncurses 6.1 he6710b0_1
        - openssl 1.1.1a h7b6447c_0
        - xz 5.2.4 h14c3975_4
        - zlib 1.2.11 h7b6447c_3
        - libedit 3.1.20170329 h6b74fdf_2
        - readline 7.0 h7b6447c_5
        - tk 8.6.8 hbc83047_0
        - sqlite 3.25.3 h7b6447c_0
        - python 3.6.7 h0371630_0
        - libfoo 1.0.0 hf82bc7d_0
outputs:
    -   name: libfoo
        requirements:
            host:
                - openssl
    -   name: foo
        requirements:
            host:
                - python
                - libfoo 1.0.0 hf82bc7d_0
extra:
    copy_test_source_files: true
    final: true
    parent_recipe:
        name: foo_split
        path: /home/jhelmus/workspace/one_offs/cb_multi_out_bug
        version: 1.0.0

Notice that in the second recipe openssl 1.1.1a is in the host section which will conflict with openssl constraint introduced by libfoo 1.0.0 via a run_export.

Building the recipe fails when solving an environment for the build of foo:

$ conda build .
...
Packaging libfoo-1.0.0-hf82bc7d_0
INFO:conda_build.build:Packaging libfoo-1.0.0-hf82bc7d_0
No files or script found for output libfoo
WARNING:conda_build.build:No files or script found for output libfoo
number of files: 0
Fixing permissions
Compressing to /tmp/tmpg2lhwuuh/libfoo-1.0.0-hf82bc7d_0.tar.bz2
Package verification results:
-----------------------------
/tmp/tmpg2lhwuuh/libfoo-1.0.0-hf82bc7d_0.tar.bz2: C1115 Found invalid license "None" in info/index.json
Packaging foo
INFO:conda_build.build:Packaging foo
Solving environment: ...working... failed

Leaving build/test directories:
  Work:
 /home/jhelmus/anaconda3/conda-bld/work 
  Test:
 /home/jhelmus/anaconda3/conda-bld/test_tmp 
Leaving build/test environments:
  Test:
source activate  /home/jhelmus/anaconda3/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold
_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p 
  Build:
source activate  /home/jhelmus/anaconda3/conda-bld/_build_env 


Traceback (most recent call last):
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda_build/environ.py", line 751, in get_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/common/io.py", line 46, in decorated
    return f(*args, **kwds)
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/plan.py", line 541, in install_actions
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/core/solve.py", line 518, in solve_for_transaction
    force_remove, force_reinstall)
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/core/solve.py", line 451, in solve_for_diff
    final_precs = self.solve_final_state(deps_modifier, prune, ignore_pinned, force_remove)
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/core/solve.py", line 344, in solve_final_state
    solution = r.solve(tuple(final_environment_specs))  # return value is List[dist]
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/common/io.py", line 46, in decorated
    return f(*args, **kwds)
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/resolve.py", line 857, in solve
    self.find_conflicts(specs)
  File "/home/jhelmus/anaconda3/lib/python3.6/site-packages/conda/resolve.py", line 244, in find_conflicts
    raise UnsatisfiableError(bad_deps)
conda.exceptions.UnsatisfiableError: The following specifications were found to be in conflict:
  - libfoo==1.0.0=hf82bc7d_0 -> openssl[version='>=1.0.2p,<1.0.3a']
  - openssl==1.1.1a=h7b6447c_0
Use "conda info <package>" to see the dependencies for each package.

[jjhelmus]

Likely related, sub-package variants are not expanded in sub-packages which have dependencies on the first sub-package.

package:
  name: bar_split
  version: 1.0.0

outputs:
  - name: libbar
    requirements:
      host:
        - hdf5

  - name: bar
    requirements:
      host:
        - python
        - {{ pin_subpackage('libbar', exact=True) }}

hdf5:
  - 1.8
  - 1.10
python:
  - 3.6

$ conda render .
read filter "zstd" is not supported
write filter "zstd" is not supported
--------------
Hash contents:
--------------
{'hdf5': '1.8'}
----------
meta.yaml:
----------
package:
    name: libbar
    version: 1.0.0
build:
    noarch: false
requirements:
    host:
        - libgcc-ng 8.2.0 hdf63c60_1
        - libgfortran-ng 7.3.0 hdf63c60_0
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - zlib 1.2.11 h7b6447c_3
        - hdf5 1.8.20 hba1933b_1
    run:
        - hdf5 >=1.8.20,<1.9.0a0
outputs:
    -   name: libbar
        requirements:
            host:
                - hdf5
    -   name: bar
        requirements:
            host:
                - python
                - libbar 1.0.0 0
extra:
    copy_test_source_files: true
    final: true
    parent_recipe:
        name: bar_split
        path: /home/jhelmus/workspace/one_offs/cb_multi_out_bug/hdf5
        version: 1.0.0


--------------
Hash contents:
--------------
{'hdf5': '1.10'}
----------
meta.yaml:
----------
package:
    name: libbar
    version: 1.0.0
build:
    noarch: false
requirements:
    host:
        - libgcc-ng 8.2.0 hdf63c60_1
        - libgfortran-ng 7.3.0 hdf63c60_0
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - zlib 1.2.11 h7b6447c_3
        - hdf5 1.10.2 hba1933b_1
    run:
        - hdf5 >=1.10.2,<1.10.3.0a0
outputs:
    -   name: libbar
        requirements:
            host:
                - hdf5
    -   name: bar
        requirements:
            host:
                - python
                - libbar 1.0.0 0
extra:
    copy_test_source_files: true
    final: true
    parent_recipe:
        name: bar_split
        path: /home/jhelmus/workspace/one_offs/cb_multi_out_bug/hdf5
        version: 1.0.0

--------------
Hash contents:
--------------
{}
----------
meta.yaml:
----------
package:
    name: bar
    version: 1.0.0
build:
    noarch: false
requirements:
    host:
        - ca-certificates 2018.03.07 0
        - libgcc-ng 8.2.0 hdf63c60_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - libffi 3.2.1 hd88cf55_4
        - ncurses 6.1 he6710b0_1
        - openssl 1.1.1a h7b6447c_0
        - xz 5.2.4 h14c3975_4
        - zlib 1.2.11 h7b6447c_3
        - libedit 3.1.20170329 h6b74fdf_2
        - readline 7.0 h7b6447c_5
        - tk 8.6.8 hbc83047_0
        - sqlite 3.25.3 h7b6447c_0
        - python 3.6.7 h0371630_0
        - libbar 1.0.0 h6fd60c2_0
outputs:
    -   name: libbar
        requirements:
            host:
                - hdf5
    -   name: bar
        requirements:
            host:
                - python
                - libbar 1.0.0 h6fd60c2_0
extra:
    copy_test_source_files: true
    final: true
    parent_recipe:
        name: bar_split
        path: /home/jhelmus/workspace/one_offs/cb_multi_out_bug/hdf5
        version: 1.0.0

Note that a single bar package will be built, rather than one for each libbar variant.

[jjacobelli]

Hey, I'm facing the same issue. Is there any update on this?

This is the reproducer I have:

package:
  name: test-split

outputs:
  - name: test-1
    version: 1
    requirements:
      run:
        - pandas >=1.0,<1.4.0dev0
  - name: test-2
    version: 1
    requirements:
      host:
        - setuptools
        - {{ pin_subpackage('test-1', exact=True) }}

While building test-2 I have a conflict because conda is trying to create a build env with test-1 which requires pandas >=1.0,<1.4.0dev0 and setuptools 62.1.0.* which is not compatible with pandas >=1.0,<1.4.0dev0.

But if I split into 2 different recipes like this:

package:
  name: test-1
  version: 1

requirements:
  run:
    - pandas >=1.0,<1.4.0dev0

package:
  name: test-2
  version: 1

requirements:
  host:
    - setuptools
    - test-1

conda is able to solve without issue and pick the right setuptools version (setuptools 59.8.0)

[jakirkham]

We discussed this earlier in the meeting today.

One suggestion was adding the shared dependency explicitly in both cases (as opposed to leaving it implicit). So tried this change to Jonathan's recipe in the OP:

diff --git a/meta.yaml b/meta.yaml
index 1dfd597..1fd17bc 100644
--- a/meta.yaml
+++ b/meta.yaml
@@ -12,4 +12,5 @@ outputs:
     requirements:
       host:
         - python
+        - openssl
         - {{ pin_subpackage('libfoo', exact=True) }}

This seem to build for me. Though dropping openssl fails as Jonathan already observed.

Also here's the result of conda render for completeness:

--------------
Hash contents:
--------------
{'openssl': '1.0.2', 'target_platform': 'osx-64'}
----------
meta.yaml:
----------
package:
  name: libfoo
  version: 1.0.0
build:
  noarch: false
  noarch_python: false
requirements:
  host:
    - ca-certificates 2021.10.8 h033912b_0
    - openssl 1.0.2u h0b31af3_0
  run:
    - openssl >=1.0.2u,<1.0.3a
extra:
  copy_test_source_files: true
  final: true
  parent_recipe:
    name: foo_split
    path: /Users/jkirkham/foo_split
    version: 1.0.0

--------------
Hash contents:
--------------
{'openssl': '1.0.2', 'target_platform': 'osx-64'}
----------
meta.yaml:
----------
package:
  name: foo
  version: 1.0.0
build:
  noarch: false
  noarch_python: false
requirements:
  host:
    - ca-certificates 2021.10.8 h033912b_0
    - libcxx 13.0.1 hc203e6f_0
    - libzlib 1.2.11 h6c3fc93_1014
    - ncurses 6.3 h96cf925_1
    - xz 5.2.5 haf1e3a3_1
    - libffi 3.2.1 hb1e8313_1007
    - openssl 1.0.2u h0b31af3_0
    - readline 7.0 hcfe32e1_1001
    - tk 8.6.12 h5dbffcc_0
    - zlib 1.2.11 h6c3fc93_1014
    - sqlite 3.28.0 h9721f7c_0
    - python 3.6.7 h4a56312_1002
    - python_abi 3.6 2_cp36m
    - wheel 0.37.1 pyhd8ed1ab_0
    - setuptools 58.0.4 py36h79c6626_2
    - pip 21.3.1 pyhd8ed1ab_0
    - libfoo 1.0.0 h2fd0e3a_0
  run:
    - openssl >=1.0.2u,<1.0.3a
extra:
  copy_test_source_files: true
  final: true
  parent_recipe:
    name: foo_split
    path: /Users/jkirkham/foo_split
    version: 1.0.0

Maybe this provides more insight as well ideas for workarounds in the interim 🙂

[skupr-anaconda]

It will work if you add this general host to the recipe https://github.com/AnacondaRecipes/modin-feedstock/blob/a9e3362d08c4cdc0601f8d31fd32dd7c3cdfe4d8/recipe/meta.yaml#L14-L19

[jakirkham]

Good to know there is a workaround. Though does it still seem fair to say this isn't working as expected?

[skupr-anaconda]

Yeah, it's not working as expected on the build CI systems but not locally. I'm not sure if it is related to the specific platforms because I can build on my osx-arm64.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!