GH-33804: [Python] Add support for manylinux_2_28 wheel #33805

sjperkins · 2023-01-20T12:30:33Z

Rationale for this change

At some point, it would be useful to support the new C++ ABI _GLIBCXX_USE_CXX11_ABI=1 in pyarrow wheels, especially when moving to C++17:

[C++] Move to C++17 #32415

I wanted to create a pyarrow wheel that supported the above ABI and adapted the existing CENTOS 7 manylinux2014 Dockerfile/wheel to produce a AlmaLinux 8 manylinux_2_28 Dockerfile/wheel.

Publishing wheels with a new ABI needs careful consideration so I think this is low priority, but I thought I'd provide this manylinux_2_28 implementation in case it was useful for current/future adoption.

What changes are included in this PR?

A manylinux_2_28 Dockerfile, adopted from the existing manylinux2014 Dockerfile

Are these changes tested?

Manually tested at present

Are there any user-facing changes?

Yes, there's a major ABI change, as pyarrow will be compiled with _GLIBCXX_USE_CXX11_ABI=1

Closes: [Packaging][Python] Add support for manylinux_2_28 wheels #33804

github-actions · 2023-01-20T12:31:00Z

Closes: [Packaging][Python] Add support for manylinux_2_28 wheels #33804

github-actions · 2023-01-20T12:31:03Z

⚠️ GitHub issue #33804 has been automatically assigned in GitHub to PR creator.

ci/docker/python-wheel-manylinux-x-y.dockerfile

kou

Could you also add entries for this to https://github.com/apache/arrow/blob/master/dev/tasks/tasks.yml#L455-L465 ?

docker-compose.yml

ci/docker/python-wheel-manylinux-x-y.dockerfile

docker-compose.yml

ci/docker/python-wheel-manylinux-x-y.dockerfile

sjperkins · 2023-01-21T11:02:48Z

Could you also add entries for this to https://github.com/apache/arrow/blob/master/dev/tasks/tasks.yml#L455-L465 ?

I've done so, but I'm not sure how to test correctness locally. This seems to be part of the crossbow archery component which kicks off nightly CI services which I'm fairly certain I don't have permissions for.

Crossbow is a subcomponent of Archery and can be used to manually trigger builds. The tasks which can be run on Crossbow can be found in the dev/tasks directory. This directory contains:

the file dev/tasks/tasks.yml containing the configuration for various tasks which can be run via Crossbow

subdirectories containing different task templates (specified using jinja2 syntax <https://jinja.palletsprojects.com/>_), divided roughly by language or package management system.

Most of these tasks are run as part of the nightly builds, though they can also be triggered manually by add a comment to a PR which begins with @github-actions crossbow submit followed by the name of the task to be run.

For convenience purpose, the tasks in dev/tasks/tasks.yml are defined in groups, which makes it simpler for multiple tasks to be submitted to Crossbow at once. The task definitions here contain information about which service defined in docker-compose.yml to run, the CI service to run the task on, and which template file to use as the basis for that task.

sjperkins · 2023-01-21T11:10:50Z

Could you also add entries for this to https://github.com/apache/arrow/blob/master/dev/tasks/tasks.yml#L455-L465 ?

I've done so, but I'm not sure how to test correctness locally. This seems to be part of the crossbow archery component which kicks off nightly CI services which I'm fairly certain I don't have permissions for.

Crossbow is a subcomponent of Archery and can be used to manually trigger builds. The tasks which can be run on Crossbow can be found in the dev/tasks directory. This directory contains:

the file dev/tasks/tasks.yml containing the configuration for various tasks which can be run via Crossbow

subdirectories containing different task templates (specified using jinja2 syntax <https://jinja.palletsprojects.com/>_), divided roughly by language or package management system.

Most of these tasks are run as part of the nightly builds, though they can also be triggered manually by add a comment to a PR which begins with @github-actions crossbow submit followed by the name of the task to be run.

For convenience purpose, the tasks in dev/tasks/tasks.yml are defined in groups, which makes it simpler for multiple tasks to be submitted to Crossbow at once. The task definitions here contain information about which service defined in docker-compose.yml to run, the CI service to run the task on, and which template file to use as the basis for that task.

I see this started CI and related tests which revealed issues, addressed in 424fa08 and bf6f3b1.

sjperkins · 2023-01-21T12:19:14Z

I don't think I understand the Archery/Crossbow setup. May I have some guidance on how to correct it please?

docker-compose.yml

ci/docker/python-wheel-manylinux-x-y.dockerfile

kou · 2023-01-21T20:44:00Z

We can run added tasks by @github-actions crossbow submit wheel-manylinux* comment after we fix "Archery & Crossbow" CI failures.
(But you can't do it yet because you're a first-time contributor for now.)

See also: https://arrow.apache.org/docs/dev/developers/continuous_integration/overview.html#action-triggered-builds

kou · 2023-01-21T20:47:25Z

You don't need to setup Crossbow locally.

dev/tasks/tasks.yml

kou · 2023-01-22T07:59:57Z

@github-actions crossbow submit wheel-manylinux*

github-actions · 2023-01-22T08:02:39Z

Revision: ee4d79d

Submitted crossbow builds: ursacomputing/crossbow @ actions-05f1584a4c

Task	Status
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp311-amd64
wheel-manylinux2014-cp311-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-manylinux_2_28-cp310-amd64
wheel-manylinux_2_28-cp310-arm64
wheel-manylinux_2_28-cp311-amd64
wheel-manylinux_2_28-cp311-arm64
wheel-manylinux_2_28-cp37-amd64
wheel-manylinux_2_28-cp37-arm64
wheel-manylinux_2_28-cp38-amd64
wheel-manylinux_2_28-cp38-arm64
wheel-manylinux_2_28-cp39-amd64
wheel-manylinux_2_28-cp39-arm64

docker-compose.yml

dev/tasks/tasks.yml

ci/docker/python-wheel-manylinux-x-y.dockerfile

Co-authored-by: Sutou Kouhei <[email protected]>

raulcd · 2023-01-23T08:49:39Z

@github-actions crossbow submit wheel-manylinux*

kou · 2023-01-30T03:14:17Z

@github-actions crossbow submit wheel-manylinux*

github-actions · 2023-01-30T03:17:10Z

Revision: 21a7ff1

Submitted crossbow builds: ursacomputing/crossbow @ actions-a5e60b310f

Task	Status
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp37-amd64
wheel-manylinux-2-28-cp37-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp37-amd64
wheel-manylinux-2014-cp37-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64

sjperkins · 2023-01-30T06:44:13Z

I still see complaints re: missing linux-headers required for openssl unfortunately so it seems dnf didn't help in this case:

https://app.travis-ci.com/github/ursacomputing/crossbow/builds/260129759#L2059

I'll investigate more locally.

sjperkins · 2023-01-30T09:47:45Z

I still see complaints re: missing linux-headers required for openssl unfortunately so it seems dnf didn't help in this case:

https://app.travis-ci.com/github/ursacomputing/crossbow/builds/260129759#L2059

If I insert the following into the Dockerfile

RUN vcpkg install \
        --clean-after-build \
        --x-install-root=${VCPKG_ROOT}/installed \
        openssl

and run

$ PYTHON=3.8 ARCH=arm64v8 archery docker run python-wheel-manylinux-2-28

it'll warn about missing linux-headers but the step will succeed.

I need to focus on other priorities for the next week. I'll try pick it up again after that. Note this error was reported on the vcpkg repo

[openssl] Build error microsoft/vcpkg#25729

where the reported solution was to install perl-IPC-Cmd (which is installed in the manylinux Dockerfile in any case).

Would it be possible to run the ci once more to get around the typo in 0cc575c?

raulcd · 2023-01-30T10:34:28Z

@github-actions crossbow submit wheel-manylinux*

github-actions · 2023-01-30T10:38:13Z

Revision: 0cc575c

Submitted crossbow builds: ursacomputing/crossbow @ actions-14d1746afa

Task	Status
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp37-amd64
wheel-manylinux-2-28-cp37-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp37-amd64
wheel-manylinux-2014-cp37-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64

sjperkins · 2023-02-24T07:16:01Z

This PR is still on my mind, but I haven't been able to devote time to working out a way forward.

sjperkins · 2023-02-24T13:14:04Z

Looks like the following issue is relevant:

[openssl] Build error with arm64-linux triplet microsoft/vcpkg#29674

sjperkins · 2023-02-24T14:53:22Z

Applied the fix 95d6827 described at the end of microsoft/vcpkg#29674 (comment):

Edit: In the port Unix/CMakeLists.txt I added "--cross-compile-prefix=\"\"" to the flags for the Perl configure custom command and now it works. I think this is a platform problem because I didn't have that problem with an ubuntu based docker image. Feel free to close if you agree.

I'm still testing this locally with a slow, emulated arm64v8 build, but it seems to have passed the openssl build step:

$ ARCH=arm64v8 PYTHON=3.8 archery docker run python-wheel-manylinux-2-28
[+] Building 4964.0s (18/22)                                                                                                                   
 => [ 4/17] COPY ci/scripts/install_cmake.sh arrow/ci/scripts/                                                                           0.0s
 => [ 5/17] RUN /arrow/ci/scripts/install_cmake.sh arm64v8 linux 3.21.4 /usr/local                                                      16.8sc 
 => [ 6/17] COPY ci/scripts/install_ninja.sh arrow/ci/scripts/                                                                           0.0s
 => [ 7/17] RUN /arrow/ci/scripts/install_ninja.sh 1.10.2 /usr/local                                                                   399.5s  
 => [ 8/17] COPY ci/scripts/install_ccache.sh arrow/ci/scripts/                                                                          0.0s
 => [ 9/17] RUN /arrow/ci/scripts/install_ccache.sh 4.1 /usr/local                                                                     368.1s  
 => [10/17] COPY ci/vcpkg/*.patch      ci/vcpkg/*linux*.cmake      arrow/ci/vcpkg/                                                       0.0s
 => [11/17] COPY ci/scripts/install_vcpkg.sh      arrow/ci/scripts/                                                                      0.0s  
 => [12/17] RUN arrow/ci/scripts/install_vcpkg.sh /opt/vcpkg 2871ddd918cecb9cb642bcb9c56897f397283192                                 1270.0s
 => [13/17] COPY ci/vcpkg/vcpkg.json arrow/ci/vcpkg/                                                                                     0.0s 
 => [14/17] RUN vcpkg install         --clean-after-build         --x-install-root=/opt/vcpkg/installed         --x-manifest-root=/a  2853.9s
 => => # -- Extracting source /opt/vcpkg/downloads/google-snappy-1.1.9.tar.gz                                                                  
 => => # -- Applying patch fix_clang-cl_build.patch                                                                                          
 => => # -- Applying patch snappy-disable-bmi.patch                                                                                            
 => => # -- Using source at /opt/vcpkg/buildtrees/snappy/src/1.1.9-405ddc5021.clean                                                          
 => => # -- Configuring arm64-linux-static-release                                                                                           
 => => # -- Building arm64-linux-static-release-rel

@kou @raulcd Could you please initiate a CI run?

sjperkins · 2023-02-24T17:02:27Z

@kou @raulcd Could you please initiate a CI run?

Ah this failed locally, nevermind.

amol- · 2023-03-30T17:06:17Z

Closing because it has been untouched for a while, in case it's still relevant feel free to reopen and move it forward 👍

sjperkins · 2023-03-30T17:46:37Z

Thanks for bumping this. Unfortunately I don't have a local arm64 system that would make it possible to quickly debug the build issues on that platform.

Perhaps if the need for this change arises in future, someone with an arm64 system could take a look.

felipecrv · 2023-03-31T01:23:52Z

@sjperkins try rebasing and push -f to get a clean CI run that might actually pass.

sjperkins · 2023-03-31T07:02:46Z

@sjperkins try rebasing and push -f to get a clean CI run that might actually pass.

I think the PR needs to re-opened for this to work? Pushes to the fork don't seem to be reflecting here anymore.

https://github.com/sjperkins/arrow/tree/manylinux-2-28

jorisvandenbossche · 2023-03-31T07:12:21Z

Pushes to the fork don't seem to be reflecting here anymore.

Because you (force) pushed while the PR was closed, it can't be reopened anymore .. Sorry! (unless you would push again the branch as it was before, https://gist.github.com/robertpainsi/2c42c15f1ce6dab03a0675348edd4e2c)

amol- · 2023-03-31T07:19:01Z

It's probably easier to just open a new PR, that wouldn't require any change on @sjperkins side and we could link the two PRs

sjperkins · 2023-03-31T07:43:40Z

It's probably easier to just open a new PR, that wouldn't require any change on @sjperkins side and we could link the two PRs

Done in #34818

Closes #33804 ### Rationale for this change At some point, it would be useful to support the new C++ ABI `_GLIBCXX_USE_CXX11_ABI=1` in pyarrow wheels, especially when moving to C++17: - #32415 I wanted to create a pyarrow wheel that supported the above ABI and adapted the existing CENTOS 7 manylinux2014 Dockerfile/wheel to produce a AlmaLinux 8 manylinux_2_28 Dockerfile/wheel. Publishing wheels with a new ABI needs [careful consideration](https://pypackaging-native.github.io/key-issues/native-dependencies/cpp_deps/) so I think this is low priority, but I thought I'd provide this manylinux_2_28 implementation in case it was useful for current/future adoption. ### What changes are included in this PR? A manylinux_2_28 Dockerfile, adopted from the existing manylinux2014 Dockerfile ### Are these changes tested? Manually tested at present ### Are there any user-facing changes? Yes, there's a major ABI change, as pyarrow will be compiled with `_GLIBCXX_USE_CXX11_ABI=1` * Closes: #33804 Supercedes: * #33805 * Closes: #33804 Lead-authored-by: Simon Perkins <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

…#34818) Closes apache#33804 ### Rationale for this change At some point, it would be useful to support the new C++ ABI `_GLIBCXX_USE_CXX11_ABI=1` in pyarrow wheels, especially when moving to C++17: - apache#32415 I wanted to create a pyarrow wheel that supported the above ABI and adapted the existing CENTOS 7 manylinux2014 Dockerfile/wheel to produce a AlmaLinux 8 manylinux_2_28 Dockerfile/wheel. Publishing wheels with a new ABI needs [careful consideration](https://pypackaging-native.github.io/key-issues/native-dependencies/cpp_deps/) so I think this is low priority, but I thought I'd provide this manylinux_2_28 implementation in case it was useful for current/future adoption. ### What changes are included in this PR? A manylinux_2_28 Dockerfile, adopted from the existing manylinux2014 Dockerfile ### Are these changes tested? Manually tested at present ### Are there any user-facing changes? Yes, there's a major ABI change, as pyarrow will be compiled with `_GLIBCXX_USE_CXX11_ABI=1` * Closes: apache#33804 Supercedes: * apache#33805 * Closes: apache#33804 Lead-authored-by: Simon Perkins <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

Add manylinux_2_28 wheel

f8fc629

sjperkins requested review from assignUser, kou and raulcd as code owners January 20, 2023 12:30

sjperkins commented Jan 20, 2023

View reviewed changes

ci/docker/python-wheel-manylinux-x-y.dockerfile Outdated Show resolved Hide resolved

sjperkins commented Jan 20, 2023

View reviewed changes

ci/docker/python-wheel-manylinux-x-y.dockerfile Outdated Show resolved Hide resolved

kou reviewed Jan 21, 2023

View reviewed changes

Address review comments

bb9268d

Fix manylinux variable discrepancies

424fa08

sjperkins added 2 commits January 21, 2023 13:25

Fix incorrect arch causing duplicate yaml keys

bf6f3b1

Avoid duplicate keys in task definition

26cc4c0

kou reviewed Jan 21, 2023

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

ci/docker/python-wheel-manylinux-x-y.dockerfile Outdated Show resolved Hide resolved

ci/docker/python-wheel-manylinux-x-y.dockerfile Outdated Show resolved Hide resolved

kou reviewed Jan 21, 2023

View reviewed changes

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

sjperkins added 2 commits January 22, 2023 07:33

fix arch and remove ci from python wheel key

b490f15

underscores to dashes in manylinux 2_28 ccache name

ee4d79d

kou reviewed Jan 22, 2023

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

ci/docker/python-wheel-manylinux-x-y.dockerfile Outdated Show resolved Hide resolved

ci/docker/python-wheel-manylinux-x-y.dockerfile Outdated Show resolved Hide resolved

sjperkins and others added 4 commits January 22, 2023 10:28

Update docker-compose.yml

d196e09

Co-authored-by: Sutou Kouhei <[email protected]>

Update dev/tasks/tasks.yml

79661a1

Co-authored-by: Sutou Kouhei <[email protected]>

Create unified manylinux Dockerfile

3363331

Remove unused install_glibc.sh

1171e32

Fix typo in artifact string

0cc575c

sjperkins added 4 commits February 2, 2023 09:31

Merge branch 'master' into manylinux-2-28

f10be24

Merge branch 'master' into manylinux-2-28

999a927

Merge branch 'main' into manylinux-2-28

dbb4473

Merge branch 'main' into manylinux-2-28

9c1a3dc

Add --cross-compile-prefix option to openssl CMakeList.txt

95d6827

amol- closed this Mar 30, 2023

sjperkins mentioned this pull request Mar 31, 2023

GH-33804: [Python] Add support for manylinux_2_28 wheel #34818

Merged

sjperkins mentioned this pull request Mar 31, 2023

GH-33801: [Python] Generate on the fly Python Extension Types wrapping C++ Extension Types #34483

Closed

GH-33804: [Python] Add support for manylinux_2_28 wheel #33805

GH-33804: [Python] Add support for manylinux_2_28 wheel #33805

Uh oh!

Conversation

sjperkins commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jan 20, 2023

Uh oh!

github-actions bot commented Jan 20, 2023

Uh oh!

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sjperkins commented Jan 21, 2023

Uh oh!

sjperkins commented Jan 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjperkins commented Jan 21, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou commented Jan 21, 2023

Uh oh!

kou commented Jan 21, 2023

Uh oh!

Uh oh!

Uh oh!

kou commented Jan 22, 2023

Uh oh!

github-actions bot commented Jan 22, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Jan 23, 2023

Uh oh!

kou commented Jan 30, 2023

Uh oh!

github-actions bot commented Jan 30, 2023

Uh oh!

sjperkins commented Jan 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjperkins commented Jan 30, 2023

Uh oh!

raulcd commented Jan 30, 2023

Uh oh!

github-actions bot commented Jan 30, 2023

Uh oh!

sjperkins commented Feb 24, 2023

Uh oh!

sjperkins commented Feb 24, 2023

Uh oh!

sjperkins commented Feb 24, 2023

Uh oh!

sjperkins commented Feb 24, 2023

Uh oh!

amol- commented Mar 30, 2023

Uh oh!

sjperkins commented Mar 30, 2023

Uh oh!

felipecrv commented Mar 31, 2023

Uh oh!

sjperkins commented Jan 20, 2023 •

edited

Loading

sjperkins commented Jan 21, 2023 •

edited

Loading

sjperkins commented Jan 30, 2023 •

edited

Loading

amol- commented Mar 31, 2023 •

edited

Loading