Distributions sagemath-{flint-arb,homfly,giac,gap}, add "sage_setup: distribution" to Cython modules

[mkoeppe]

Basically, we are making all external libraries optional, except for a minimal set of "core" libraries such as cysignals and gmpy2 (see #29865).

(from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sagemath-ntl) - but some libraries come in packs that would not make much sense to separate (such as sagemath-flint-arb-e_antic).

List of distributions:

• sagemath-flint-arb
  - 'sage/libs/arb/arb_version.pyx',
  - 'sage/libs/arb/arith.pyx',
  - 'sage/matrix/matrix_complex_ball_dense.pyx',
  - 'sage/rings/complex_arb.pyx',
  - 'sage/rings/number_field/number_field_element_quadratic.pyx',
  - 'sage/rings/polynomial/polynomial_complex_arb.pyx',
  - 'sage/rings/real_arb.pyx'
  - etc.
• sagemath-homfly: 'sage/libs/homfly.pyx'
• sagemath-giac (ex giacpy_sage in sage.libs.giac + dependency on libgiac)
  - sage/libs/giac/giac.pyx
• sagemath-gap:
  - 'sage/coding/codecan/codecan.pyx',
  - 'sage/combinat/enumeration_mod_permgroup.pyx',
  - 'sage/combinat/root_system/reflection_group_c.pyx',
  - 'sage/combinat/root_system/reflection_group_element.pyx',
  - 'sage/graphs/spanning_tree.pyx',
  - 'sage/groups/libgap_wrapper.pyx',
  - 'sage/groups/matrix_gps/group_element.pyx',
  - 'sage/groups/perm_gps/permgroup_element.pyx',
  - 'sage/groups/perm_gps/partn_ref/automorphism_group_canonical_label.pyx',
  - 'sage/groups/perm_gps/partn_ref/canonical_augmentation.pyx',
  - 'sage/groups/perm_gps/partn_ref/data_structures.pyx',
  - 'sage/groups/perm_gps/partn_ref/double_coset.pyx',
  - 'sage/groups/perm_gps/partn_ref/refinement_binary.pyx',
  - 'sage/groups/perm_gps/partn_ref/refinement_graphs.pyx',
  - 'sage/groups/perm_gps/partn_ref/refinement_lists.pyx',
  - 'sage/groups/perm_gps/partn_ref/refinement_matrices.pyx',
  - 'sage/groups/perm_gps/partn_ref/refinement_python.pyx',
  - 'sage/groups/perm_gps/partn_ref/refinement_sets.pyx',
  - 'sage/groups/perm_gps/partn_ref2/refinement_generic.pyx',
  - 'sage/libs/gap/element.pyx',
  - 'sage/libs/gap/libgap.pyx',
  - 'sage/libs/gap/util.pyx',
  - 'sage/matrix/matrix_gap.pyx',
  - 'sage/sets/disjoint_set.pyx'

To test:

• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_flint_arb
• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_gap
• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_giac
• make SAGE_WHEELS=yes SAGE_CHECK=yes sagemath_homfly

In follow-up tickets:

• sagemath-brial (ex sage-brial in sage.rings.polynomial.pbori + sage.libs.polybori, sage.crypto.boolean_function)
• sagemath-ntl
• sagemath-ecl (sage.libs.ecl)
• sagemath-maxima: requires sage-ecl
• sagemath-singular: (some of these files actually may be better put in the pynac distribution)
  - 'sage/algebras/letterplace/free_algebra_element_letterplace.pyx',
  - 'sage/algebras/letterplace/free_algebra_letterplace.pyx',
  - 'sage/algebras/letterplace/letterplace_ideal.pyx',
  - 'sage/libs/pynac/constant.pyx',
  - 'sage/libs/pynac/pynac.pyx',
  - 'sage/libs/singular/function.pyx',
  - 'sage/libs/singular/groebner_strategy.pyx',
  - 'sage/libs/singular/option.pyx',
  - 'sage/libs/singular/polynomial.pyx',
  - 'sage/libs/singular/ring.pyx',
  - 'sage/libs/singular/singular.pyx',
  - 'sage/matrix/matrix_mpolynomial_dense.pyx',
  - 'sage/rings/polynomial/multi_polynomial_ideal_libsingular.pyx',
  - 'sage/rings/polynomial/multi_polynomial_libsingular.pyx',
  - 'sage/rings/polynomial/plural.pyx',
  - 'sage/symbolic/comparison.pyx',
  - 'sage/symbolic/constants_c.pyx',
  - 'sage/symbolic/expression.pyx',
  - 'sage/symbolic/function.pyx',
  - 'sage/symbolic/getitem.pyx',
  - 'sage/symbolic/ring.pyx',
  - 'sage/symbolic/series.pyx',
  - 'sage/symbolic/substitution_map.pyx',
• sagemath-pynac
• sagemath-linbox:
  - 'sage/libs/linbox/linbox_flint_interface.pyx',
  - 'sage/matrix/matrix_integer_sparse.pyx',
  - 'sage/matrix/matrix_modn_dense_double.pyx',
  - 'sage/matrix/matrix_modn_dense_float.pyx',
  - 'sage/matrix/matrix_modn_sparse.pyx'
• sagemath-znpoly (subject to removal in zn_poly removal #32841?)
  - 'sage/modular/modsym/p1list.pyx',
  - 'sage/modular/pollack_stevens/dist.pyx',
  - 'sage/rings/fraction_field_FpT.pyx',
  - 'sage/rings/polynomial/polynomial_zmod_flint.pyx',
  - 'sage/schemes/hyperelliptic_curves/hypellfrob.pyx',
• sagemath-mari (also requiring pbori and pbori-groebner):
  - 'sage/matrix/matrix_gf2e_dense.pyx',
  - 'sage/matrix/matrix_integer_dense.pyx',
  - 'sage/matrix/matrix_mod2_dense.pyx',
  - 'sage/matrix/matrix_rational_dense.pyx',
  - 'sage/modules/vector_mod2_dense.pyx',
  - 'sage/rings/polynomial/pbori/pbori.pyx',
  - 'sage/rings/polynomial/polynomial_gf2x.pyx'

This is preparation for Meta-ticket #29705 (Modularization) and #30371.

Depends on #31031
Depends on #34855
Depends on #34839

CC: @tobiasdiez @dimpase

Component: refactoring

Keywords: sd111

Author: Matthias Koeppe

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30666

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,2 +1,6 @@
+... except for a minimal set of libraries such as `cysignals` and `gmpy2` (see #29865).
+
+(from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sage-ntl) - but some libraries come in packs that would not make much sense to separate (such as sage-flint-arb-e_antic).
+
 This is preparation for Meta-ticket #29705 (Modularization) and #30371.
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -2,5 +2,15 @@
 
 (from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sage-ntl) - but some libraries come in packs that would not make much sense to separate (such as sage-flint-arb-e_antic).
 
+List of distributions:
+
+- `sage-giac` (ex `giacpy_sage` in `sage.libs.giac` + dependency on `libgiac`)
+- `sage-brial` (ex `sage-brial` in `sage.rings.polynomial.pbori` +  `sage.libs.polybori`, `sage.crypto.boolean_function`)
+- `sage-ntl`
+- `sage-flint-arb-e_antic`
+- `sage-ecl` (`sage.libs.ecl`)
+- `sage-singular`
+- `sage-pynac`
+
 This is preparation for Meta-ticket #29705 (Modularization) and #30371.
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-... except for a minimal set of libraries such as `cysignals` and `gmpy2` (see #29865).
+... except for a minimal set of "core" libraries such as `cysignals` and `gmpy2` (see #29865), which would require 
 
 (from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sage-ntl) - but some libraries come in packs that would not make much sense to separate (such as sage-flint-arb-e_antic).
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-... except for a minimal set of "core" libraries such as `cysignals` and `gmpy2` (see #29865), which would require 
+Basically, we are making all external libraries optional, except for a minimal set of "core" libraries such as `cysignals` and `gmpy2` (see #29865). 
 
 (from #30371): See #29705 under the items titled "Deploy mildly modularized distributions", "Further modularization" for a sketch of such distributions. Basically, one distribution for each major C/C++ library (such as sage-ntl) - but some libraries come in packs that would not make much sense to separate (such as sage-flint-arb-e_antic).
 

[tobiasdiez]

comment:5

Good idea to separate this into a new ticket.

While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.

Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.

What do you think about renaming sage_setup: distribution to sage_setup: dependency? I think this better reflects the purpose. The distribution is then generated based on which dependencies are available. Another idea would be to use distutils: libraries which already contains some of the dependencies anyway (e.g. arb, gmp, flint).

[mkoeppe]

comment:6

Replying to @tobiasdiez:

While working on #30371, I noticed that most cython files using singular cannot be compiled without pynac. So maybe these should be combined.

I think they should be separate because I really want to be able to make singular optional. pynac, on the other hand, is essential for our symbolics implementation.

Also, I think, there should be distributions for GAP, CCObject, linbox, ratpoints, zn_poly and mari.

Yes, please add to the list in the ticket description with explanation (what is CCObject?)

What do you think about renaming sage_setup: distribution to sage_setup: dependency?

No, I wouldn't like that. As soon as you start with declaring "dependencies", there will be combinatorial explosion from the combinations of such dependencies. I really want a well-designed partition of the source files into (relatively few) disjoint distributions. This is crucial for the modularization -- each distribution will eventually be separately compilable and installable package on PyPI.

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -11,6 +11,11 @@
 - `sage-ecl` (`sage.libs.ecl`)
 - `sage-singular`
 - `sage-pynac`
+- `sage-gap` (see #30371 setup.py for which files need GAP)
+- `sage-linbox` (see #30371 setup.py for which files need linbox)
+- `sage-ratpoints` (see #30371 setup.py for which files need ratpoints)
+- `sage-znpoly` (see #30371 setup.py for which files need zn_poly)
+- `sage-mari` (see #30371 setup.py for which files need mari)
 
 This is preparation for Meta-ticket #29705 (Modularization) and #30371.
 

[tobiasdiez]

comment:8

Ok, makes sense.
Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?

My idea was that the cython file declares which dependencies it has, and in the setup.py file you have a list of distributions and their dependencies. If all dependencies are met, that particular distribution is built. E.g. distribution "sage-singular" relies on "pynac" and "singular". (That's only a feeling, but it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sage-singular" vs "sage-symbolic calculation". But you have a better overview, so I trust you there.)

I've added the above distributions to the list.
(Forget about CCObject, it was a missing import in #30371 that confused me.)

[mkoeppe]

comment:9

Replying to @tobiasdiez:

Which distribution should be assigned to files that have two dependencies (e.g use pynac and singular)?

Distributions can certainly depend on other distributions. sage-singular could depend on sage-pynac. Let's include this in the ticket description. (In the modularized build, sage-singular's setup.py or setup.cfg would have install_requires = sage-pynac.)

Also, often it is relatively easy to remove some of these dependencies by small changes to a few source files, moving a few methods around. See for example #29911. We can take work on this in follow-up tickets.

it feels strange to have distributions determined by dependencies and not by a theme or subject, e.g. "sage-singular" vs "sage-symbolic calculation".

Perhaps, but there is actually an important "political" (open source community) reason for this design too. By naming these components after major libraries that provide the features, we make these libraries and their important role for Sage more visible to end users. Attribution tends to be an important motivating factor to developers of these libraries.

[mkoeppe]

comment:10

To expand on this a little bit: I wouldn't rule out distributions named after topics. The present ticket only provides a "skeleton" of distributions that are (1) technically needed to take care of compile-time dependencies, (2) provides attribution. Later on (in follow-up tickets), for example, there could be a distribution sage-feature-padics or something like this that has sage-ntl as a dependency and packages sage.rings.padics.

[tobiasdiez]

Description changed:

--- 
+++ 
@@ -8,14 +8,87 @@
 - `sage-brial` (ex `sage-brial` in `sage.rings.polynomial.pbori` +  `sage.libs.polybori`, `sage.crypto.boolean_function`)
 - `sage-ntl`
 - `sage-flint-arb-e_antic`
+      - 'sage/libs/arb/arith.pyx',
+      - 'sage/matrix/matrix_complex_ball_dense.pyx',
+      - 'sage/rings/complex_arb.pyx',
+      - 'sage/rings/number_field/number_field_element_quadratic.pyx',
+      - 'sage/rings/polynomial/polynomial_complex_arb.pyx',
+      - 'sage/rings/real_arb.pyx'
 - `sage-ecl` (`sage.libs.ecl`)
-- `sage-singular`
+- `sage-singular`: (some of these files actually may be better put in the pynac distribution)
+      - 'sage/algebras/letterplace/free_algebra_element_letterplace.pyx',
+      - 'sage/algebras/letterplace/free_algebra_letterplace.pyx',
+      - 'sage/algebras/letterplace/letterplace_ideal.pyx',
+      - 'sage/libs/pynac/constant.pyx',
+      - 'sage/libs/pynac/pynac.pyx',
+      - 'sage/libs/singular/function.pyx',
+      - 'sage/libs/singular/groebner_strategy.pyx',
+      - 'sage/libs/singular/option.pyx',
+      - 'sage/libs/singular/polynomial.pyx',
+      - 'sage/libs/singular/ring.pyx',
+      - 'sage/libs/singular/singular.pyx',
+      - 'sage/matrix/matrix_mpolynomial_dense.pyx',
+      - 'sage/rings/polynomial/multi_polynomial_ideal_libsingular.pyx',
+      - 'sage/rings/polynomial/multi_polynomial_libsingular.pyx',
+      - 'sage/rings/polynomial/plural.pyx',
+      - 'sage/symbolic/comparison.pyx',
+      - 'sage/symbolic/constants_c.pyx',
+      - 'sage/symbolic/expression.pyx',
+      - 'sage/symbolic/function.pyx',
+      - 'sage/symbolic/getitem.pyx',
+      - 'sage/symbolic/ring.pyx',
+      - 'sage/symbolic/series.pyx',
+      - 'sage/symbolic/substitution_map.pyx',
 - `sage-pynac`
-- `sage-gap` (see #30371 setup.py for which files need GAP)
-- `sage-linbox` (see #30371 setup.py for which files need linbox)
-- `sage-ratpoints` (see #30371 setup.py for which files need ratpoints)
+- `sage-gap`:
+      - 'sage/coding/codecan/codecan.pyx',
+      - 'sage/combinat/enumeration_mod_permgroup.pyx',
+      - 'sage/combinat/root_system/reflection_group_c.pyx',
+      - 'sage/combinat/root_system/reflection_group_element.pyx',
+      - 'sage/graphs/spanning_tree.pyx',
+      - 'sage/groups/libgap_wrapper.pyx',
+      - 'sage/groups/matrix_gps/group_element.pyx',
+      - 'sage/groups/perm_gps/permgroup_element.pyx',
+      - 'sage/groups/perm_gps/partn_ref/automorphism_group_canonical_label.pyx',
+      - 'sage/groups/perm_gps/partn_ref/canonical_augmentation.pyx',
+      - 'sage/groups/perm_gps/partn_ref/data_structures.pyx',
+      - 'sage/groups/perm_gps/partn_ref/double_coset.pyx',
+      - 'sage/groups/perm_gps/partn_ref/refinement_binary.pyx',
+      - 'sage/groups/perm_gps/partn_ref/refinement_graphs.pyx',
+      - 'sage/groups/perm_gps/partn_ref/refinement_lists.pyx',
+      - 'sage/groups/perm_gps/partn_ref/refinement_matrices.pyx',
+      - 'sage/groups/perm_gps/partn_ref/refinement_python.pyx',
+      - 'sage/groups/perm_gps/partn_ref/refinement_sets.pyx',
+      - 'sage/groups/perm_gps/partn_ref2/refinement_generic.pyx',
+      - 'sage/libs/gap/element.pyx',
+      - 'sage/libs/gap/libgap.pyx',
+      - 'sage/libs/gap/util.pyx',
+      - 'sage/matrix/matrix_gap.pyx',
+      - 'sage/sets/disjoint_set.pyx'
+- `sage-linbox`:
+      - 'sage/libs/linbox/linbox_flint_interface.pyx',
+      - 'sage/matrix/matrix_integer_sparse.pyx',
+      - 'sage/matrix/matrix_modn_dense_double.pyx',
+      - 'sage/matrix/matrix_modn_dense_float.pyx',
+      - 'sage/matrix/matrix_modn_sparse.pyx'
+- `sage-ratpoints`: 'sage/libs/ratpoints.pyx', 'sage/schemes/elliptic_curves/descent_two_isogeny.pyx'
 - `sage-znpoly` (see #30371 setup.py for which files need zn_poly)
-- `sage-mari` (see #30371 setup.py for which files need mari)
+      - 'sage/modular/modsym/p1list.pyx',
+      - 'sage/modular/pollack_stevens/dist.pyx',
+      - 'sage/rings/fraction_field_FpT.pyx',
+      - 'sage/rings/polynomial/polynomial_zmod_flint.pyx',
+      - 'sage/schemes/hyperelliptic_curves/hypellfrob.pyx',
+- `sage-mari` (also requiring pbori and pbori-groebner):
+      - 'sage/matrix/matrix_gf2e_dense.pyx',
+      - 'sage/matrix/matrix_integer_dense.pyx',
+      - 'sage/matrix/matrix_mod2_dense.pyx',
+      - 'sage/matrix/matrix_rational_dense.pyx',
+      - 'sage/modules/vector_mod2_dense.pyx',
+      - 'sage/rings/polynomial/pbori/pbori.pyx',
+      - 'sage/rings/polynomial/polynomial_gf2x.pyx'
+- `sage-magma`: requires sage-ecl
+- `sage-giac`: 'sage/libs/giac/giac.pyx'
+- `sage-homfly`: 'sage/libs/homfly.pyx'
 
 This is preparation for Meta-ticket #29705 (Modularization) and #30371.