Reject unsupported extern "{abi}"s consistently in all positions

[workingjubilee]

Modify the handling of extern "{abi}" in the compiler so that it has consistent errors without regard to the position in the grammar.

What

Implement the following breakages:

• Promote unsupported_fn_ptr_calling_conventions from a warning to a hard error
• Guarantee future edge-cases like trait declarations will not escape so that ABIs that have already been unusable in most positions will now be unusable in all positions. See "How" and "Why or Why Not" for more details.

In particular, these architecture-specific ABIs now only compile on their architectures¹:

• amdgpu: "gpu-kernel"
• arm: "aapcs", "C-cmse-nonsecure-entry", "C-cmse-nonsecure-call"
• avr: "avr-interrupt", "avr-non-blocking-interrupt"
• msp430: "msp430-interrupt"
• nvptx64: "gpu-kernel", "ptx-kernel"
• riscv32 and riscv64: "riscv-interrupt-m", "riscv-interrupt-s"
• x86: "thiscall"
• x86 and x86_64: "x86-interrupt"
• x86_64: "sysv64", "win64"

ABIs that are logically x86-specific but actually permitted on all Windows targets remain permitted on Windows, as before. For non-Windows targets, they error if we had previously done so in other positions.

How

We modify rustc_ast_lowering to prevent unsupported ABIs from leaking through the HIR without being checked for target support. They now emit hard errors for every case where we would return Invalid from AbiMap::canonize_abi. Previously ad-hoc checking on various HIR items required making sure we check every HIR item which could contain an extern "{abi}" string. This is a losing proposition compared to gating the lowering itself.

As a consequence, unsupported ABI strings error instead of triggering the warning unsupported_fn_ptr_calling_conventions. The code is also simpler compared to alternative implementations that might e.g. split on unstable vs. stable, only suffering some unavoidable complication to support the newly-revived unsupported_calling_conventions lint.²

However, per #86232 this does cause errors for rare usages of extern "{abi}" that were theoretically possible to write in Rust source, without previous warning or error. For instance, trait declarations without impls were never checked. These are the exact kinds of leakages that this new approach prevents.

This differs from the following PRs:

• Add (back) unsupported_calling_conventions lint to reject more invalid calling conventions #141435 is orthogonal, as it adds a new lint for ABIs we have not warned on and are not touched by this PR
• Reject unsupported, unstable ABIs during AST lowering #141877 is subsumed by this, in that this simply cuts out bad functionality instead of adding epicycles for stable code

Why or Why Not

We already made the decision to issue the unsupported_fn_ptr_calling_conventions future compatibility warning. It has warned in dependencies since #135767, which reached stable with Rust 1.87. That was released on 2025 May 17, and it is now June. As we already had erred on these ABI strings in most other positions, and warn on stable for function pointer types, this breakage has had reasonable foreshadowing.

Upgrading the warning to an error addresses a real problem. In some cases the Rust compiler can attempt to actually compute the ABI for calling a function with an unsupported ABI. We could accept this case and compute unsupported ABIs according to some other ABI, silently³. However, this obviously exposes Rust to errors in codegen. We cannot lower directly to the "obvious", target-incorrect ABI and then trust code generators like LLVM to reliably error on these cases, either.

Other considerations include:

• We could refactor the compiler to defer ABI computations, but that seems like it would suffer the "whack-a-mole" problem close to linking instead of after parsing and expansion.
• We could run a deprecation cycle for the edge cases, but we would be warning highly marginal cases, like this trait declaration without a definition that cannot be implemented without error⁴.

pub trait UsedToSneakBy {
    pub extern "gpu-kernel" fn sneaky();
}

• The crater run on this PR's draft form suggests the primary issue is with implementations on function pointers, which has already been warned on, so it does not seem like we would be benefiting any real code.
• Converting this to a hard error now, in the same cycle that we ship the reanimation of the unsupported_calling_conventions lint, means people who would otherwise have to deal with two lints only have to update their code in one batch. Of course, one of them is as breakage.

r? lang

Fixes #86232
Fixes #132430
Fixes #138738
Fixes #142107

Footnotes

1. Some already will not compile, due to reaching ICEs or LLVM errors. ↩

2. That lint cannot be moved in a similar way yet because lints operate on HIR, so you cannot emit lints when the HIR has not been completely formed. ↩

3. We already do this for all AbiStr we cannot parse, pretending they are ExternAbi::Rust, but we also emit an error to prevent reaching too far into codegen. ↩

4. Upon any impl, even for provided fn within trait declarations, e.g. pub extern "gpu-kernel" fn sneaky() {}, different HIR types were used which would, in fact, get checked. Likewise for anything with function pointers. Thus we would be discussing deprecation cycles for code that is impotent or forewarned⁵. ↩

5. It actually did appear in two cases in rustc's test suite because we are a collection of Rust edge-cases by the simple fact that we don't care if the code actually runs. These cases are being excised in 643a9d2 ↩

[rustbot]

HIR ty lowering was modified

cc @fmease

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

These commits modify tests/rustdoc-json.
rustdoc-json is a public (but unstable) interface.

Please ensure that if you've changed the output:

• It's intentional.
• The FORMAT_VERSION in src/librustdoc-json-types is bumped if necessary.

cc @aDotInTheVoid, @obi1kenobi

[workingjubilee]

@bors2 try

[rust-bors]

⌛ Trying commit c1db989 with merge 8b8eff5…

To cancel the try build, run the command @bors2 try cancel.

[rust-bors]

☀️ Try build successful (CI)
Build commit: 8b8eff5 (8b8eff55bd72abbb57167bc42222a7f91d41cb0d)

[workingjubilee]

@craterbot run mode=build-only name=pr-142134-abi-ast-error cap-lints=warn

[craterbot]

🚧 Experiment pr-142134-abi-ast-error is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

👌 Experiment pr-142134-abi-ast-error created and queued.
🤖 Automatically detected try build 8b8eff5
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-142134-abi-ast-error is completed!
📊 57 regressed and 16 fixed (643383 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[workingjubilee]

I'll send the necessary PRs to size-of and retour.

The primary culprit seems to be impls on function pointers, which are already warned on: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=e1e8ca91878cbbcba5c6dc3b064cea2b

on retour

I should note that while impls on function pointers are convenient, it's not clear that code that works at the low level that retour does can be correct when interacting with these function pointers unless it is written with full awareness that the relevant ABIs are translated to extern "C". Some simple versions, maybe, but often detouring libraries have to make assumptions about codegen, and consistently getting that correct seems unlikely. So I believe enforcing this lint in a now-absolutist fashion will, in fact, make the world's code more correct.

on size_of

For size_of the upgrade to warn-on-deps seems to have attracted notice: Kixiron/size-of#6

I was able to contact the maintainer who is happy to run a release once I draft a PR that fixes this.

on the rest

• https://github.com/EnmanuelParache/msp430fr2476 is archived, but clearly only intended to be a crossbuild for msp430 devices which would support msp430-interrupt in the first place.
• https://github.com/ejaanens/msp430fr2476 is a skeleton project generated by svd2rust and has the same caveat.
• https://github.com/bekker/qvopenapi-rs looks abandoned and I expect a contrib will have a language barrier, but I'll try.

The "regressions" for libc, serde_derive, etc. all appear to be spurious failures due to build space running out.

[bors]

☔ The latest upstream changes (presumably #141435) made this pull request unmergeable. Please resolve the merge conflicts.

[RalfJung]

Fixes #86232

So what does this do with invalid ABIs in traits? I can't see a new lint being introduced here -- but making this a hard error immediately would be a breaking change, no?

[rustbot]

This PR modifies tests/auxiliary/minicore.rs.

cc @jieyouxu

[RalfJung]

Regarding the FIXME just above this line: since #86232 will be marked as fixed, it seems good to update the FIXME. Github doesn't think this is useful though so we have to do it by hand:

// FIXME: this currently only guards function imports, function definitions, and function pointer types.
// Functions in trait declarations can still use "deprecated" ABIs without any warning.

[RalfJung]

The HIR parts LGTM, but I know nothing about that AST code so I can't tell if that's truly the one place all ABI strings go through.