Ubuntu 22.04 20250223.1.0 breaking aarch64

[zhangskz]

Description

Three of our aarch64 CI tests running on the Ubuntu 22.04 large runners started failing / flaking coinciding with the 20250223.1.0 release.

These were broken with the same problem a ~month ago after a Ubuntu 20.04 image release, and we originally worked around it by upgrading to Ubuntu 22.04:
protocolbuffers/protobuf@06ba410

We suspect that this is a kernel bug in the Ubuntu image (shared kernel that Docker also uses, e.g. for the Ruby aarch64 failure) since it looks like the kernel version did change in this release.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

20250223.1.0

Examples (three different aarch64 presubmits):
https://github.com/protocolbuffers/protobuf/actions/runs/13575835948/job/37951791581
https://github.com/protocolbuffers/protobuf/actions/runs/13574908994/job/37948793310
https://github.com/protocolbuffers/protobuf/actions/runs/13574908994/job/37948786669

Is it regression?

Yes, 20250209.1.0 e.g. https://github.com/protocolbuffers/protobuf/actions/runs/13545933121/job/37857479819

Expected behavior

Tests should pass

Actual behavior

Segmentation fault (C++), error running bundle, when run using github runner images that don't reproduce from physical machine.

[ 81%] Running cpp protocol buffer compiler on /workspace/src/google/protobuf/unittest_import.proto
gmake[2]: *** [CMakeFiles/libtest_common.dir/build.make:114: google/protobuf/map_unittest.pb.h] Segmentation fault (core dumped)
gmake[2]: *** Waiting for unfinished jobs....
[ 81%] Built target upb-test

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/ffi-1.17.1/ext/ffi_c
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.0.0 -r
./siteconf20250227-19-zw7v2f.rb extconf.rb
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

/workspace/php/ext/google/protobuf/protobuf.c:270:(.text+0xadc): dangerous relocation: unsupported relocation
/bin/ld: .libs/protobuf.o: relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `protobuf_module_entry' which may bind externally can not be used when making a shared object; recompile with -fPIC
.libs/protobuf.o: in function `get_module':
/workspace/php/ext/google/protobuf/protobuf.c:323:(.text+0xb68): dangerous relocation: unsupported relocation
collect2: error: ld returned 1 exit status
make: *** [Makefile:244: protobuf.la] Error 1

Repro steps

From ubuntu-22-4core image, I believe something like the following should repro:

$ git clone https://github.com/protocolbuffers/protobuf.git
$ cd protobuf
~/protobuf $ docker run -it --entrypoint=bash arm64v8/ruby:3.0.2-buster
root@8bb9c96f191d:/workspace# gem install bundler -v 2.5.13;
root@8bb9c96f191d:/workspace# cd /workspace/ruby;
root@8bb9c96f191d:/workspace# bundle;

See https://github.com/protocolbuffers/protobuf/blob/87a2b4645b44f458e60ee5cf7b2a49541909b133/.github/workflows/test_ruby.yml#L126