Qemu fails since ubuntu-24.04 20250202.1.0

[kohtala]

Description

Issues #11471, #11557 and #11561 have been closed on false claim it would be a partner image issue. The partner-runner-image repo confirms this is not in their image: actions/partner-runner-images#52

Builds segfault on the x64 GitHub runner image.

I find there is tonistiigi/binfmt#215 on the qemu being used, but this issue appeared without a change to qemu on an ubuntu-24.04 image upgrade from 20250126.1.0 to 20250202.1.0. While they have done some study on qemu and Linux kernel patches, they have been unable to find a fix on the qemu to get this working.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

20250202.1.0 https://github.com/okoko/qpid-cpp-docker/actions/runs/13233057744/job/36933224760
20250209.1.0 https://github.com/okoko/qpid-cpp-docker/actions/runs/13517577090/job/37769409633

Is it regression?

Yes. 20250126.1.0 https://github.com/okoko/qpid-cpp-docker/actions/runs/13105871226

Expected behavior

The build to pass without segmentation fault. Like it used to

Current runner version: '2.322.0'
Operating System
  Ubuntu
  24.04.1
  LTS
Runner Image
  Image: ubuntu-24.04
  Version: 20250126.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250126.1/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250126.1
Runner Image Provisioner
  2.0.422.1
...
/usr/bin/docker image inspect docker.io/tonistiigi/binfmt:latest
  [
      {
          "Id": "sha256:354472a378935adfe74a19600b89bd9ada7bb058306fff23b3d6613405852faf",
          "RepoTags": [
              "tonistiigi/binfmt:latest"
          ],
          "RepoDigests": [
              "tonistiigi/binfmt@sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55"
          ],
...
#78 exporting to image
#78 pushing layers 1.5s done
#78 pushing manifest for docker.io/okoko/qpid-cpp:1.39.0-0.40.0@sha256:7ef0fb0a182cb4e6b2a254a25b94cb856201011dc3a9c947483695c339d38084
#78 pushing manifest for docker.io/okoko/qpid-cpp:1.39.0-0.40.0@sha256:7ef0fb0a182cb4e6b2a254a25b94cb856201011dc3a9c947483695c339d38084 2.2s done
#78 pushing layers 0.3s done
#78 pushing manifest for docker.io/okoko/qpid-cpp:latest@sha256:7ef0fb0a182cb4e6b2a254a25b94cb856201011dc3a9c947483695c339d38084
#78 pushing manifest for docker.io/okoko/qpid-cpp:latest@sha256:7ef0fb0a182cb4e6b2a254a25b94cb856201011dc3a9c947483695c339d38084 0.8s done
#78 DONE 7.3s

Actual behavior

Regardless of the version of qemu, the build fails with segmentation fault.

Build with recently updated tonistiigi/binfmt:latest build has

Current runner version: '2.322.0'
Operating System
  Ubuntu
  24.04.1
  LTS
Runner Image
  Image: ubuntu-24.04
  Version: 20250209.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250209.1/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250209.1
Runner Image Provisioner
  2.0.422.1
...
/usr/bin/docker image inspect docker.io/tonistiigi/binfmt:latest
  [
      {
          "Id": "sha256:a74106e88df18d1354a77767d7475f3d778bfb49a73962778d4e4898228f3295",
          "RepoTags": [
              "tonistiigi/binfmt:latest"
          ],
          "RepoDigests": [
              "tonistiigi/binfmt@sha256:ea2f0dd74e74f101df59f9a6b31d0960994060c7982a921cbceecee0f1841125"
          ],
...
83.98 Processing triggers for libc-bin (2.31-13+deb11u11) ...
84.08 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
84.75 Segmentation fault (core dumped)
84.79 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
85.53 Segmentation fault (core dumped)
85.54 dpkg: error processing package libc-bin (--configure):
85.55  installed libc-bin package post-installation script subprocess returned error exit status 139
85.68 Errors were encountered while processing:
85.68  libc-bin
86.48 E: Sub-process /usr/bin/dpkg returned an error code (1)

Build with the same tonistiigi/binfmt:latest image as in the latest successful build, but newer ubuntu-24.04 runner image:

Current runner version: '2.322.0'
Operating System
  Ubuntu
  24.04.1
  LTS
Runner Image
  Image: ubuntu-24.04
  Version: 20250202.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250202.1/images/ubuntu/Ubuntu2404-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250202.1
Runner Image Provisioner
  2.0.422.1
...
  /usr/bin/docker image inspect docker.io/tonistiigi/binfmt:latest
  [
      {
          "Id": "sha256:354472a378935adfe74a19600b89bd9ada7bb058306fff23b3d6613405852faf",
          "RepoTags": [
              "tonistiigi/binfmt:latest"
          ],
          "RepoDigests": [
              "tonistiigi/binfmt@sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55"
          ],
...
#50 284.9 [ 10%] Building CXX object src/CMakeFiles/qpidcommon.dir/qpid/framing/ExecutionExceptionBody.cpp.o
#50 285.1 c++: internal compiler error: Segmentation fault signal terminated program cc1plus
#50 285.1 Please submit a full bug report,

Repro steps

1. Take a workflow performing multiplatform docker build using qemu
2. Run it

[kishorekumar-anchala]

Thank you for bringing this issue to our attention. We will look into this issue and will update you after investigating.


[ankudinov]

Just to add a bit of context. There are number of threads around docker/setup-qemu-action, tonistiigi/binfmt image, buildx, etc. discussing this. The most useful comment I have seen so far is: docker/buildx#2841 (comment)
Looks like tonistiigi/binfmt:qemu-v7.0.0-28 is failing as well starting today. I'm curios if that can be related to kernel changes in the latest Ubuntu 24.04 runner image?

[subir0071]

Hello @kohtala - Executed the workflow shared in one of the mentioned related issue #11561 . The workflow completed successfully.
From this ticket description it looks like the issue is with Qemu.

Let me know if I am missing something.
Or, how to modify the sample workflow to reproduce the error.
Thanks.

[kohtala]

Builds that had consistently succeeded started to consistently fail with the same qemu binary on runner-image upgrade from 20250126.1.0 to 20250202.1.0. Since the problem appeared when nothing changed in qemu, but at the runner-image change, I find it unlikely it is caused by qemu.

qemu has since been upgraded and the time the failure happens changed, but nevertheless it still fails. The same qemu build works on workstations running the upstream Ubuntu kernel.

Only difference between failures and successes I find is in the kernel.

@ankudinov pointed, it might be in the buildx.
Last successful build used github.com/docker/buildx v0.20.1 245093b99ab74aa2c729a496759afca0704d6470.
First failed build ran the same github.com/docker/buildx v0.20.1 245093b99ab74aa2c729a496759afca0704d6470.
So it seems unlikely to suddenly appear in buildx without a change in it.

I suspect the build performed in the #11561 is too small. I tried my workflow just a while ago and get the segmentation fault consistently now when it's installing libc-bin under qemu:

 > [linux/arm64 build  2/15] RUN --mount=type=cache,target=/var/cache/apt,id=bullseye-/var/cache/apt-arm64     --mount=type=cache,target=/var/lib/apt,sharing=locked,id=bullseye-/var/lib/apt-arm64     <<NUR (set -ex...):
90.99 Processing triggers for libc-bin (2.31-13+deb11u11) ...
91.12 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
91.81 Segmentation fault (core dumped)
91.85 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
92.61 Segmentation fault (core dumped)
92.61 dpkg: error processing package libc-bin (--configure):
92.62  installed libc-bin package post-installation script subprocess returned error exit status 139
92.75 Errors were encountered while processing:
92.75  libc-bin
93.51 E: Sub-process /usr/bin/dpkg returned an error code (1)
------

I'll try to return to testing this with only this libc-bin install. I suspect if you setup the qemu from tonistiigi/binfmt@sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55, your test workflow might hit the problem in the old position, where builds used to succeed with 20250126.1.0 and before.

But unfortunately I now have to go and try to return to this later.

[saltydk]

https://gitlab.com/qemu-project/qemu/-/issues/1913 might be related, its fix is in release 9.2.2

[ankudinov]

tonistiigi/binfmt@sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55 (aka tonistiigi/binfmt:qemu-v7.0.0-28) is not a consistent fix. I've seen the workflows failing yesterday with this exact image and segfault error. This working again later same day. I guess we are catching these errors as we build multiple times a day.

Part of the challenge is that it's not possible to rollback to old image on public Github runners to test old kernel configuration. Anyone running self-hosted runners is certainly in a better position.

There is another great thread here: tonistiigi/binfmt#215 (comment)
https://gitlab.com/qemu-project/qemu/-/issues/1913 mentioned by @saltydk can be related and I'd wait for 9.2.2 to be integrated into tonistiigi/binfmt

[kohtala]

Since the issue mentions ldconfig, which indeed can be segfaulting during libc-bin setup, I checked if the ldconfig is any different in the builds. The docker.io/library/buildpack-deps:bullseye has been different in each build, but the ldconfig binary was exactly the same in all buildpack-deps images.

Last successful build

#13 [linux/arm64 build  1/15] FROM docker.io/library/buildpack-deps:bullseye@sha256:5a732df6c8a203396d95ce485ee02ede5840e312cdd5bdd60b51155633cc234e
#13 resolve docker.io/library/buildpack-deps:bullseye@sha256:5a732df6c8a203396d95ce485ee02ede5840e312cdd5bdd60b51155633cc234e done
#13 DONE 0.0s

First failed build

#23 [linux/arm64 build  1/15] FROM docker.io/library/buildpack-deps:bullseye@sha256:23f9292b468d7a21f9dc077e08fd09ad8f39d08ab44841da9b7f5877dc50e5dc
#23 resolve docker.io/library/buildpack-deps:bullseye@sha256:23f9292b468d7a21f9dc077e08fd09ad8f39d08ab44841da9b7f5877dc50e5dc done
#23 sha256:2cae5cdedf5465f9b64d9afa67592e36727e50a811baf4a0d35508744cd435c3 189.98MB / 189.98MB 1.2s done
#23 sha256:bf8e2f45c7ddf8cc116eeb2ac1ef8be70e3639a883c6d9e5eaf1f2dd702dbf92 54.85MB / 54.85MB 0.5s done
#23 sha256:82c7afb1aa0f9672a06c4c7eaa6b7c7e111a91a8d45272dce1e361ac0b0ed79a 15.54MB / 15.54MB 0.2s done
#23 sha256:0038ef039a89ede34c57806e684dc9d9be7dcd22d3c08b90deb36bb22a2c7122 52.25MB / 52.25MB 1.3s done
#23 extracting sha256:0038ef039a89ede34c57806e684dc9d9be7dcd22d3c08b90deb36bb22a2c7122
#23 ...

Last failed build

#23 [linux/arm64 build  1/15] FROM docker.io/library/buildpack-deps:bullseye@sha256:df8d2cddbecb9c8843657a4b8b4f2688ebd46e3188737cb5cf59c1f7cbeb74ff
#23 extracting sha256:eb2bd61bef017e8e3b4e6f403c074fa47f738d2ac56d92eb50f50fff5dc8bd03 4.0s done
#23 DONE 9.8s

$ regctl image get-file --platform arm64 buildpack-deps@sha256:5a732df6c8a203396d95ce485ee02ede5840e312cdd5bdd60b51155633cc234e /sbin/ldconfig ldconfig-5a732df6c8a203396d95ce485ee02ede5840e312cdd5bdd60b51155633cc234e
...
$ sha256sum ldconfig-*
fb08339e541908e6e9b78d3cf67b23d304d81f839b72e340de7d890670ac5ca3  ldconfig-23f9292b468d7a21f9dc077e08fd09ad8f39d08ab44841da9b7f5877dc50e5dc
fb08339e541908e6e9b78d3cf67b23d304d81f839b72e340de7d890670ac5ca3  ldconfig-5a732df6c8a203396d95ce485ee02ede5840e312cdd5bdd60b51155633cc234e
fb08339e541908e6e9b78d3cf67b23d304d81f839b72e340de7d890670ac5ca3  ldconfig-df8d2cddbecb9c8843657a4b8b4f2688ebd46e3188737cb5cf59c1f7cbeb74ff

I also tested if the current tonistiigi/binfmt:master@sha256:e88b685d6032025f19ee41951a6c63c8ccdc3467d0b226cf4ff0c4e95ebab286 that was built just little after the patches were picked up, but at least it still segfaulted just the same as earlier images. But I don't know for sure if the patches were in the build.

[TingDaoK]

We used ubuntu-22.04 as a workaround before,
but looks like since 20250223.1.0, ubuntu-22.04 has the same issue now.
Segfault from gcc when qume used.

  Image: ubuntu-22.04
  Version: 20250223.1.0