aarch64 builds using nightly fail with missing `getauxval` in `init_have_lse_atomics` in `cpu_model.c` in `compiler_builtins`

[tonyg]

[Updated] Workaround

See this comment.

To reproduce

1. mkdir foo
2. cd foo
3. cargo init
4. cross build --target=aarch64-unknown-linux-musl --release --all-targets -- note that this succeeds
5. echo 'nightly' > rust-toolchain
6. cross build --target=aarch64-unknown-linux-musl --release --all-targets -- note that this fails

Actual outcome

Step 6 fails with

$ cross build --target=aarch64-unknown-linux-musl --release --all-targets
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'

  nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.57.0-nightly (485ced56b 2021-10-07)

info: checking for self-updates
   Compiling foo v0.1.0 (/project)
error: linking with `aarch64-linux-musl-gcc` failed: exit status: 1
  |
  = note: "aarch64-linux-musl-gcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.foo.4b08363c-cgu.0.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.foo.4b08363c-cgu.1.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.foo.4b08363c-cgu.2.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.6er9z6ghb47uqlg.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-unknown-linux-musl/release/deps" "-L" "/target/release/deps" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-c44788b657ddddd5.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-2becdccda730ce42.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libminiz_oxide-3a7898da562cf313.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libadler-fc70ddb33855e2e8.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libobject-5fb8217618711e88.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libmemchr-c061be432db3e13d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libaddr2line-5a6d99ea33f6ceae.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libgimli-6db2c8017781e41d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd_detect-7d0d2f13c64acb61.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_demangle-77c2cc6c5af8536c.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libhashbrown-8ca99a806b6c7500.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_alloc-eced44336e84fd45.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libunwind-ec4fe46000190b79.rlib" "-lunwind" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcfg_if-692ed2e2b4273428.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-72b57ab90668a509.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc-7aed2b0a9d39bb09.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_core-e0db88e40d9c7e0b.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcore-fcedc0d4b8cb02ca.rlib" "-Wl,--end-group" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-nostartfiles" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-lgcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: /rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib(cpu_model.o): In function `init_have_lse_atomics':
          /cargo/registry/src/github.jpy.wang-1ecc6299db9ec823/compiler_builtins-0.1.49/./lib/builtins/cpu_model.c:786: undefined reference to `getauxval'
          collect2: error: ld returned 1 exit status

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `foo` due to previous error
warning: build failed, waiting for other jobs to finish...
error: linking with `aarch64-linux-musl-gcc` failed: exit status: 1
  |
  = note: "aarch64-linux-musl-gcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.foo.b81dabbf-cgu.0.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.foo.b81dabbf-cgu.1.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.foo.b81dabbf-cgu.2.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.11e6h28jheunbtcu.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-unknown-linux-musl/release/deps" "-L" "/target/release/deps" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libtest-3a3779ef37dcf104.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libgetopts-4b9c52f1a6321a32.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libunicode_width-3a27e1f0db71671d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_std-9ce51e2ad117da5c.rlib" "-Wl,--start-group" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-c44788b657ddddd5.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-2becdccda730ce42.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libminiz_oxide-3a7898da562cf313.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libadler-fc70ddb33855e2e8.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libobject-5fb8217618711e88.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libmemchr-c061be432db3e13d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libaddr2line-5a6d99ea33f6ceae.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libgimli-6db2c8017781e41d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd_detect-7d0d2f13c64acb61.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_demangle-77c2cc6c5af8536c.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libhashbrown-8ca99a806b6c7500.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_alloc-eced44336e84fd45.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libunwind-ec4fe46000190b79.rlib" "-lunwind" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcfg_if-692ed2e2b4273428.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-72b57ab90668a509.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc-7aed2b0a9d39bb09.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_core-e0db88e40d9c7e0b.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcore-fcedc0d4b8cb02ca.rlib" "-Wl,--end-group" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-nostartfiles" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-lgcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: /rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib(cpu_model.o): In function `init_have_lse_atomics':
          /cargo/registry/src/github.jpy.wang-1ecc6299db9ec823/compiler_builtins-0.1.49/./lib/builtins/cpu_model.c:786: undefined reference to `getauxval'
          collect2: error: ld returned 1 exit status

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: build failed

Expected outcome

Step 6 should succeed :-)

[tonyg]

(For reference, --target=armv7-unknown-linux-musleabihf builds work, even with nightly.)

[tonyg]

OK, after rather a lot of flailing about, I've managed to get foo to build with nightly. TL;DR: I added musl's libc.a to the rustc invocation via ... -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a. I did this manually by looking at the -vv output of cross and running adjusted docker commands interactively; now I need to figure out how to do it automatically!

---

Here's a (whitespace-adjusted reformatted-to-multiline-with-backslashes) version of the command that cargo is running inside the docker container:

PATH=$PATH:/rust/bin \
CARGO=/rust/bin/cargo \
CARGO_BIN_NAME=foo \
CARGO_CRATE_NAME=foo \
CARGO_MANIFEST_DIR=/project \
CARGO_PKG_AUTHORS='' \
CARGO_PKG_DESCRIPTION='' \
CARGO_PKG_HOMEPAGE='' \
CARGO_PKG_LICENSE='' \
CARGO_PKG_LICENSE_FILE='' \
CARGO_PKG_NAME=foo \
CARGO_PKG_REPOSITORY='' \
CARGO_PKG_VERSION=0.1.0 \
CARGO_PKG_VERSION_MAJOR=0 \
CARGO_PKG_VERSION_MINOR=1 \
CARGO_PKG_VERSION_PATCH=0 \
CARGO_PKG_VERSION_PRE='' \
CARGO_PRIMARY_PACKAGE=1 \
LD_LIBRARY_PATH='/target/debug/deps:/rust/lib' \
rustc \
--crate-name foo \
--edition=2018 \
src/main.rs \
--error-format=json \
--json=diagnostic-rendered-ansi \
--crate-type bin \
--emit=dep-info,link \
-C embed-bitcode=no \
-C debuginfo=2 \
-C metadata=cba3bae5335ad02d \
-C extra-filename=-cba3bae5335ad02d \
--out-dir /target/aarch64-unknown-linux-musl/debug/deps \
--target aarch64-unknown-linux-musl \
-C linker=aarch64-linux-musl-gcc \
-C incremental=/target/aarch64-unknown-linux-musl/debug/incremental \
-L dependency=/target/aarch64-unknown-linux-musl/debug/deps \
-L dependency=/target/debug/deps \
-C link-arg=-lgcc

To this I added the following, at the end:

... \
-C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a

So, the whole docker invocation that successfully built me an executable nightly foo was:

"/usr/bin/docker" "run" \
    "--userns" "host" \
    "-e" "LDFLAGS" \
    "-e" "PKG_CONFIG_ALLOW_CROSS=1" \
    "--rm" \
    "--user" "1000:1000" \
    "-e" "XARGO_HOME=/xargo" \
    "-e" "CARGO_HOME=/cargo" \
    "-e" "CARGO_TARGET_DIR=/target" \
    "-e" "USER=tonyg" \
    "-e" "CROSS_RUNNER=" \
    "-v" "/home/tonyg/.xargo:/xargo:Z" \
    "-v" "/home/tonyg/.cargo:/cargo:Z" \
    "-v" "/cargo/bin" \
    "-v" "/home/tonyg/nobackup/build/issue-598/foo:/project:Z" \
    "-v" "/home/tonyg/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu:/rust:Z,ro" \
    "-v" "/home/tonyg/nobackup/build/issue-598/foo/target:/target:Z" \
    "-w" "/project" \
    "-i" "rustembedded/cross:aarch64-unknown-linux-musl-0.2.1" \
    "sh" \
    "-c" \
    "PATH=$PATH:/rust/bin CARGO=/rust/bin/cargo CARGO_BIN_NAME=foo CARGO_CRATE_NAME=foo CARGO_MANIFEST_DIR=/project CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=foo CARGO_PKG_REPOSITORY='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH='/target/debug/deps:/rust/lib' rustc --crate-name foo --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=cba3bae5335ad02d -C extra-filename=-cba3bae5335ad02d --out-dir /target/aarch64-unknown-linux-musl/debug/deps --target aarch64-unknown-linux-musl -C linker=aarch64-linux-musl-gcc -C incremental=/target/aarch64-unknown-linux-musl/debug/incremental -L dependency=/target/aarch64-unknown-linux-musl/debug/deps -L dependency=/target/debug/deps -C link-arg=-lgcc -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a"

(The last long line there is just a one-line version of the rustc command just above.)

So, looking forward:

1. Maybe there could be a workaround involving somehow arranging for -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a to be given to rustc through the whole cross-to-docker-to-cargo-to-rustc chain?
2. What changed recently to cause this problem? It's not clear to me whether cross or cargo or something else is at fault.

[tonyg]

(Also: for debugging this I have found commands like cross -vv build -vv ... to be useful, with verbose turned on both for cross and for cargo)

[tonyg]

Automated Workaround

In Cross.toml:

[build.env]
passthrough = [
    "RUSTFLAGS",
    "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS",
]

and then

CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a" \
    cross -vv build -vv --target=aarch64-unknown-linux-musl

Interesting points to note:

1. I didn't put -lgcc in there. Setting RUSTFLAGS seems to override the -lgcc that is put in there by default. Could it be that, for musl builds, -lgcc isn't needed? Certainly for foo it seems to be OK, but maybe more complex programs still need it? Building with RUSTFLAGS="-C link-arg=-lgcc -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a" also works.

2. There's both RUSTFLAGS and CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS in Cross.toml. It doesn't work without both, if you want to use CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS! If you're happy using plain RUSTFLAGS, then having just RUSTFLAGS in Cargo.toml works fine.

An alternative workaround

Courtesy of @hkratz here, is to use RUSTFLAGS="-Zgcc-ld=lld". You'll still need the Cross.toml above, but with an alternative RUSTFLAGS setting:

CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-Zgcc-ld=lld" \
    cross build --target=aarch64-unknown-linux-musl

Relevant links

• discussion of issue: Undefined reference to getauxval in function init_have_lse_atomics when compiling to nightly aarch64-unknown-linux-musl rust-lang/rust#89626
• potential temporary revert of the problematic commit that enabled +outline-atomics: Restrict aarch64 outline atomics to glibc for now. rust-lang/rust#90044

[tonyg]

So one possible general (?) fix might be to modify aarch64-linux-musl-gcc.sh to put in the path to libc, the way it already special-cases -lgcc for older toolchains, but is that the right thing to do? Is it even a toolchain version change that caused the issue? There seem like an awful lot of moving pieces, and all I know is that roughly a week ago this started happening overnight. I'm not sure what the correct test for adding in a special-casing of libc would be.

[Emilgardis]

Related to rust-lang/libc#2450, this is probably an issue in libc and not cross

[Emilgardis]

Does this issue disappear if you pin libc to say 0.2.102 ?

[tonyg]

Does this issue disappear if you pin libc to say 0.2.102 ?

No. If I add a dependency libc = "=0.2.102" to Cargo.toml in a fresh foo project (as above), with nightly in rust-toolchain, I get the same error about undefined reference to getauxval. (I'm skeptical that the libc crate is to blame, but I definitely don't have deep expertise here -- doesn't the fact that this occurs in a fresh project without any dependencies mean that the libc crate isn't even present?)

[Emilgardis]

(I'm skeptical that the libc crate is to blame, but I definitely don't have deep expertise here -- doesn't the fact that this occurs in a fresh project without any dependencies mean that the libc crate isn't even present?)

You are correct, I missed that the project didn't use libc, just assumed so. My bad!

The issue is now tracked in rust-lang/rust#89626 , closing this issue.