GH-46746: [C++] Assume AWS SDK >= 1.11.0 #46742

pitrou · 2025-06-09T07:35:18Z

Rationale for this change

We're currently carrying compatibility code for AWS SDK before 1.9.0. However, we almost always bundled our own build of AWS SDK, so we should be able to rely on a more recent version. Version 1.11.0 was released in January 2023, which is already 2.5 years ago, so this should be a reasonable target.

This will make the S3 filesystem code more maintainable.

Are these changes tested?

Yes, on existing CI builds.

Are there any user-facing changes?

Users won't be able to build Arrow C++ if they have a very old version of the AWS SDK installed, unless they pass -DAWSSDK_SOURCE=BUNDLED.

GitHub Issue: [C++] Assume AWS SDK >= 1.11.0 #46746

github-actions · 2025-06-09T07:35:45Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

pitrou · 2025-06-09T07:56:05Z

@github-actions crossbow submit -g cpp -g python

pitrou · 2025-06-09T07:56:16Z

@github-actions crossbow submit wheelcp313*

pitrou · 2025-06-09T07:56:22Z

@github-actions crossbow submit -g r

pitrou · 2025-06-09T09:19:12Z

CI failures are unrelated.

pitrou · 2025-06-09T09:42:48Z

@github-actions crossbow submit -g cpp -g python

pitrou · 2025-06-09T09:43:02Z

@github-actions crossbow submit wheelcp313*

pitrou · 2025-06-09T09:43:12Z

@github-actions crossbow submit -g r

pitrou · 2025-06-09T09:47:03Z

Oh, I had missed that our bundled version is still 1.10.55 (!). It was released in January 2023...

pitrou · 2025-06-09T09:52:32Z

I'm switching to draft until #45306 is merged.

pitrou · 2025-06-18T07:48:55Z

@github-actions crossbow submit -g cpp -g python

pitrou · 2025-06-18T07:49:04Z

@github-actions crossbow wheelcp313*

pitrou · 2025-06-18T07:49:08Z

@github-actions crossbow submit -g r

github-actions · 2025-06-18T07:49:46Z

No such command '*wheel*cp313*'.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/15727028902

pitrou · 2025-06-18T07:51:18Z

@github-actions crossbow submit wheelcp313*

pitrou · 2025-06-18T13:37:17Z

@github-actions crossbow submit -g r

kou

+1

cpp/src/arrow/filesystem/s3fs.cc

pitrou · 2025-06-24T07:29:06Z

@github-actions crossbow submit -g cpp -g python

pitrou · 2025-06-24T07:29:16Z

@github-actions crossbow submit wheelcp313*

github-actions · 2025-06-24T07:31:54Z

Revision: bcbeaf9

Submitted crossbow builds: ursacomputing/crossbow @ actions-c8b603b831

Task	Status
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-amd64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-manylinux-2014-cp313-cp313-amd64
wheel-manylinux-2014-cp313-cp313-arm64
wheel-manylinux-2014-cp313-cp313t-amd64
wheel-manylinux-2014-cp313-cp313t-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313t-amd64
wheel-musllinux-1-2-cp313-cp313t-arm64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64

github-actions · 2025-06-24T07:32:29Z

Revision: bcbeaf9

Submitted crossbow builds: ursacomputing/crossbow @ actions-0e0f175ff0

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-cpp
test-fedora-39-python-3
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer
test-ubuntu-24.04-python-3

kou · 2025-06-24T11:31:55Z

cpp/src/arrow/filesystem/s3fs.cc

+// Keep this in sync with ThirdPartyToolChain.cmake
+#if !defined(AWS_SDK_VERSION_MAJOR) || !ARROW_AWS_SDK_VERSION_CHECK(1, 11, 0)
+#  error "AWS SDK version 1.11.0 or later is required"
 #endif


We can remove this because we have the same check in CMake. CMake rejects older AWS SDK for C++. So we don't need to check this here.

I thought this might be useful for Meson.

Ah. @WillAyd Could you remove this when you add AWS SDK for C++ support to our Meson configuration?

No problem - that and GCS support are likely some of the last things to be done for full Meson coverage, so we have some time :-)

conbench-apache-arrow · 2025-06-24T14:48:39Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 85b4e6d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

github-actions bot added Component: C++ awaiting review Awaiting review labels Jun 9, 2025

This comment was marked as outdated.

Sign in to view

pitrou added the CI: Extra Run extra CI label Jun 9, 2025

pitrou changed the title ~~EXP: [C++] Assume AWS SDK >= 1.11.0~~ GH-46746: [C++] Assume AWS SDK >= 1.11.0 Jun 9, 2025

pitrou marked this pull request as ready for review June 9, 2025 08:56

pitrou force-pushed the aws-sdk-ver branch from 51fdc99 to 9448f14 Compare June 9, 2025 09:42

This comment was marked as outdated.

Sign in to view

pitrou marked this pull request as draft June 9, 2025 09:52

pitrou force-pushed the aws-sdk-ver branch from 9448f14 to 33cfd2d Compare June 18, 2025 07:42

pitrou closed this Jun 18, 2025

pitrou reopened this Jun 18, 2025

This comment was marked as outdated.

Sign in to view

pitrou marked this pull request as ready for review June 18, 2025 12:49

pitrou requested a review from kou June 18, 2025 12:49

This comment was marked as outdated.

Sign in to view

kou approved these changes Jun 19, 2025

View reviewed changes

cpp/src/arrow/filesystem/s3fs.cc Outdated Show resolved Hide resolved

github-actions bot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Jun 19, 2025

[C++] Assume AWS SDK >= 1.11.0

bcbeaf9

pitrou force-pushed the aws-sdk-ver branch from 33cfd2d to bcbeaf9 Compare June 24, 2025 07:28

pitrou merged commit 85b4e6d into apache:main Jun 24, 2025
36 of 38 checks passed

pitrou removed the awaiting merge Awaiting merge label Jun 24, 2025

pitrou mentioned this pull request Jun 24, 2025

[C++] Assume AWS SDK >= 1.11.0 #46746

Closed

pitrou deleted the aws-sdk-ver branch June 24, 2025 09:02

kou reviewed Jun 24, 2025

View reviewed changes

github-actions bot added the awaiting changes Awaiting changes label Jun 24, 2025

GH-46746: [C++] Assume AWS SDK >= 1.11.0 #46742

GH-46746: [C++] Assume AWS SDK >= 1.11.0 #46742

Uh oh!

Conversation

pitrou commented Jun 9, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jun 9, 2025

Uh oh!

pitrou commented Jun 9, 2025

Uh oh!

pitrou commented Jun 9, 2025

Uh oh!

pitrou commented Jun 9, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

pitrou commented Jun 9, 2025

Uh oh!

pitrou commented Jun 9, 2025

Uh oh!

pitrou commented Jun 9, 2025

Uh oh!

pitrou commented Jun 9, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

pitrou commented Jun 9, 2025

Uh oh!

pitrou commented Jun 9, 2025

Uh oh!

pitrou commented Jun 18, 2025

Uh oh!

pitrou commented Jun 18, 2025

Uh oh!

pitrou commented Jun 18, 2025

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

pitrou commented Jun 18, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

pitrou commented Jun 18, 2025

Uh oh!

This comment was marked as outdated.

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pitrou commented Jun 24, 2025

Uh oh!

pitrou commented Jun 24, 2025

Uh oh!

github-actions bot commented Jun 24, 2025

Uh oh!

github-actions bot commented Jun 24, 2025

Uh oh!

Uh oh!

kou Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

pitrou Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

kou Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

WillAyd Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

conbench-apache-arrow bot commented Jun 24, 2025

Uh oh!

Uh oh!

pitrou commented Jun 9, 2025 •

edited by github-actions bot

Loading