Skip to content

Commit 21d7baa

Browse files
sjperkinskou
authored and
ArgusLi
committed
apacheGH-33804: [Python] Add support for manylinux_2_28 wheel (apache#34818)
Closes apache#33804 ### Rationale for this change At some point, it would be useful to support the new C++ ABI `_GLIBCXX_USE_CXX11_ABI=1` in pyarrow wheels, especially when moving to C++17: - apache#32415 I wanted to create a pyarrow wheel that supported the above ABI and adapted the existing CENTOS 7 manylinux2014 Dockerfile/wheel to produce a AlmaLinux 8 manylinux_2_28 Dockerfile/wheel. Publishing wheels with a new ABI needs [careful consideration](https://pypackaging-native.github.io/key-issues/native-dependencies/cpp_deps/) so I think this is low priority, but I thought I'd provide this manylinux_2_28 implementation in case it was useful for current/future adoption. ### What changes are included in this PR? A manylinux_2_28 Dockerfile, adopted from the existing manylinux2014 Dockerfile ### Are these changes tested? Manually tested at present ### Are there any user-facing changes? Yes, there's a major ABI change, as pyarrow will be compiled with `_GLIBCXX_USE_CXX11_ABI=1` * Closes: apache#33804 Supercedes: * apache#33805 * Closes: apache#33804 Lead-authored-by: Simon Perkins <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
1 parent 6da504b commit 21d7baa

File tree

7 files changed

+57
-56
lines changed

7 files changed

+57
-56
lines changed

‎.env

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,8 +96,8 @@ DEVTOOLSET_VERSION=
9696
# Please also update the crossbow configuration in order to keep the github
9797
# actions cache up to date for the macOS wheels:
9898
# https://github.com/ursacomputing/crossbow/blob/master/.github/workflows/cache_vcpkg.yml
99-
# vcpkg minimum version "09adfdc8cdad76345b7cc7f3305899e1cbd66297" due to CVE-2022-3786
100-
VCPKG="2871ddd918cecb9cb642bcb9c56897f397283192"
99+
# TODO: Update to an official release tag https://github.com/apache/arrow/issues/35099
100+
VCPKG="b619a233fbf7b2c9765fb4458f3ecb05bd3166e3" # 2023.04.03
101101

102102
# This must be updated when we update
103103
# ci/docker/python-wheel-windows-vs2017.dockerfile.

‎ci/docker/python-wheel-manylinux-201x.dockerfile renamed to ‎ci/docker/python-wheel-manylinux.dockerfile

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,19 @@ ARG manylinux
2424

2525
ENV MANYLINUX_VERSION=${manylinux}
2626

27+
# Ensure dnf is installed, especially for the manylinux2014 base
28+
RUN yum install -y dnf
29+
2730
# Install basic dependencies
28-
RUN yum install -y git flex curl autoconf zip perl-IPC-Cmd wget
31+
RUN dnf install -y git flex curl autoconf zip perl-IPC-Cmd wget kernel-headers
32+
33+
# A system Python is required for ninja and vcpkg in this Dockerfile.
34+
# On manylinux2014 base images, system Python is 2.7.5, while
35+
# on manylinux_2_28, no system python is installed.
36+
# We therefore override the PATH with Python 3.8 in /opt/python
37+
# so that we have a consistent Python version across base images.
38+
ENV CPYTHON_VERSION=cp38
39+
ENV PATH=/opt/python/${CPYTHON_VERSION}-${CPYTHON_VERSION}/bin:${PATH}
2940

3041
# Install CMake
3142
# AWS SDK doesn't work with CMake=3.22 due to https://gitlab.kitware.com/cmake/cmake/-/issues/22524
@@ -45,12 +56,10 @@ RUN /arrow/ci/scripts/install_ccache.sh ${ccache} /usr/local
4556

4657
# Install vcpkg
4758
ARG vcpkg
48-
ARG glibc=2.18
4959
COPY ci/vcpkg/*.patch \
5060
ci/vcpkg/*linux*.cmake \
5161
arrow/ci/vcpkg/
5262
COPY ci/scripts/install_vcpkg.sh \
53-
ci/scripts/install_glibc.sh \
5463
arrow/ci/scripts/
5564
ENV VCPKG_ROOT=/opt/vcpkg
5665
RUN arrow/ci/scripts/install_vcpkg.sh ${VCPKG_ROOT} ${vcpkg}
@@ -77,6 +86,7 @@ RUN vcpkg install \
7786
--x-feature=json \
7887
--x-feature=parquet
7988

89+
# Configure Python for applications running in the bash shell of this Dockerfile
8090
ARG python=3.8
8191
ENV PYTHON_VERSION=${python}
8292
RUN PYTHON_ROOT=$(find /opt/python -name cp${PYTHON_VERSION/./}-*) && \

‎ci/scripts/install_glibc.sh

Lines changed: 0 additions & 42 deletions
This file was deleted.

‎dev/release/verify-release-candidate.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1015,7 +1015,7 @@ test_linux_wheels() {
10151015
fi
10161016

10171017
local python_versions="${TEST_PYTHON_VERSIONS:-3.7m 3.8 3.9 3.10 3.11}"
1018-
local platform_tags="manylinux_2_17_${arch}.manylinux2014_${arch}"
1018+
local platform_tags="${TEST_WHEEL_PLATFORM_TAGS:-manylinux_2_17_${arch}.manylinux2014_${arch} manylinux_2_28_${arch}}"
10191019

10201020
for python in ${python_versions}; do
10211021
local pyver=${python/m}

‎dev/tasks/python-wheels/github.linux.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ jobs:
6868
-e TEST_DEFAULT=0 \
6969
-e TEST_PYARROW_VERSION={{ arrow.no_rc_version }} \
7070
-e TEST_PYTHON_VERSIONS={{ python_version }} \
71+
-e TEST_WHEEL_PLATFORM_TAGS={{ wheel_platform_tag }} \
7172
-e TEST_WHEELS=1 \
7273
almalinux-verify-rc
7374
@@ -82,6 +83,7 @@ jobs:
8283
-e TEST_DEFAULT=0 \
8384
-e TEST_PYARROW_VERSION={{ arrow.no_rc_version }} \
8485
-e TEST_PYTHON_VERSIONS={{ python_version }} \
86+
-e TEST_WHEEL_PLATFORM_TAGS={{ wheel_platform_tag }} \
8587
-e TEST_WHEELS=1 \
8688
ubuntu-verify-rc
8789
@@ -96,6 +98,7 @@ jobs:
9698
-e TEST_DEFAULT=0 \
9799
-e TEST_PYARROW_VERSION={{ arrow.no_rc_version }} \
98100
-e TEST_PYTHON_VERSIONS={{ python_version }} \
101+
-e TEST_WHEEL_PLATFORM_TAGS={{ wheel_platform_tag }} \
99102
-e TEST_WHEELS=1 \
100103
ubuntu-verify-rc
101104

‎dev/tasks/tasks.yml

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -416,17 +416,21 @@ tasks:
416416

417417
{############################## Wheel Linux ##################################}
418418

419-
{% for arch, arch_alias, x_y, manylinux in [("amd64", "x86_64", "2_17", "2014"),
420-
("arm64", "aarch64", "2_17", "2014")] %}
421-
wheel-manylinux{{ manylinux }}-{{ python_tag }}-{{ arch }}:
419+
420+
{% for arch, manylinux_version, platform_tag in [("amd64", "2014", "manylinux_2_17_x86_64.manylinux2014_x86_64"),
421+
("amd64", "2-28", "manylinux_2_28_x86_64"),
422+
("arm64", "2014", "manylinux_2_17_aarch64.manylinux2014_aarch64"),
423+
("arm64", "2-28", "manylinux_2_28_aarch64")] %}
424+
wheel-manylinux-{{ manylinux_version }}-{{ python_tag }}-{{ arch }}:
422425
ci: github
423426
template: python-wheels/github.linux.yml
424427
params:
425428
arch: "{{ arch }}"
426429
python_version: "{{ python_version }}"
427-
manylinux_version: "{{ manylinux }}"
430+
manylinux_version: "{{ manylinux_version }}"
431+
wheel_platform_tag: "{{ platform_tag }}"
428432
artifacts:
429-
- pyarrow-{no_rc_version}-{{ python_tag }}-{{ abi_tag }}-manylinux_{{ x_y }}_{{ arch_alias }}.manylinux{{ manylinux }}_{{ arch_alias }}.whl
433+
- pyarrow-{no_rc_version}-{{ python_tag }}-{{ abi_tag }}-{{ platform_tag }}.whl
430434
{% endfor %}
431435

432436
{############################## Wheel OSX ####################################}
@@ -1438,13 +1442,13 @@ tasks:
14381442
- docs.tar.gz
14391443

14401444
############################## CUDA tests #################################
1441-
1445+
14421446
test-cuda-cpp:
14431447
ci: github
14441448
template: docker-tests/github.cuda.yml
14451449
params:
14461450
image: ubuntu-cuda-cpp
1447-
1451+
14481452
test-cuda-python:
14491453
ci: github
14501454
template: docker-tests/github.cuda.yml

‎docker-compose.yml

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@ x-hierarchy:
161161
- postgres
162162
- python-wheel-manylinux-2014:
163163
- java-jni-manylinux-2014
164+
- python-wheel-manylinux-2-28
164165
- python-wheel-manylinux-test-imports
165166
- python-wheel-manylinux-test-unittests
166167
- python-wheel-windows-vs2017
@@ -183,6 +184,8 @@ volumes:
183184
name: maven-cache
184185
python-wheel-manylinux2014-ccache:
185186
name: python-wheel-manylinux2014-ccache
187+
python-wheel-manylinux-2-28-ccache:
188+
name: python-wheel-manylinux-2-28-ccache
186189
python-wheel-windows-clcache:
187190
name: python-wheel-windows-clcache
188191
ubuntu-ccache:
@@ -1019,7 +1022,7 @@ services:
10191022
python: ${PYTHON}
10201023
manylinux: 2014
10211024
context: .
1022-
dockerfile: ci/docker/python-wheel-manylinux-201x.dockerfile
1025+
dockerfile: ci/docker/python-wheel-manylinux.dockerfile
10231026
cache_from:
10241027
- ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2014-vcpkg-${VCPKG}
10251028
environment:
@@ -1029,6 +1032,29 @@ services:
10291032
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux2014-ccache:/ccache:delegated
10301033
command: /arrow/ci/scripts/python_wheel_manylinux_build.sh
10311034

1035+
# See available versions at:
1036+
# https://quay.io/repository/pypa/manylinux_2_28_x86_64?tab=tags
1037+
python-wheel-manylinux-2-28:
1038+
image: ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2-28-vcpkg-${VCPKG}
1039+
build:
1040+
args:
1041+
arch: ${ARCH}
1042+
arch_short: ${ARCH_SHORT}
1043+
base: quay.io/pypa/manylinux_2_28_${ARCH_ALIAS}:2023-01-14-103cb93
1044+
vcpkg: ${VCPKG}
1045+
python: ${PYTHON}
1046+
manylinux: 2_28
1047+
context: .
1048+
dockerfile: ci/docker/python-wheel-manylinux.dockerfile
1049+
cache_from:
1050+
- ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2-28-vcpkg-${VCPKG}
1051+
environment:
1052+
<<: *ccache
1053+
volumes:
1054+
- .:/arrow:delegated
1055+
- ${DOCKER_VOLUME_PREFIX}python-wheel-manylinux-2-28-ccache:/ccache:delegated
1056+
command: /arrow/ci/scripts/python_wheel_manylinux_build.sh
1057+
10321058
python-wheel-manylinux-test-imports:
10331059
image: ${ARCH}/python:${PYTHON}
10341060
shm_size: 2G

0 commit comments

Comments
 (0)