Skip to content

The check_abi function in typeck is not called for ABIs in trait declarations #86232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nagisa opened this issue Jun 11, 2021 · 1 comment · May be fixed by #142134
Open

The check_abi function in typeck is not called for ABIs in trait declarations #86232

nagisa opened this issue Jun 11, 2021 · 1 comment · May be fixed by #142134
Labels
A-ABI Area: Concerning the application binary interface (ABI) C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@nagisa
Copy link
Member

nagisa commented Jun 11, 2021

Consider the following code:

trait T {
    extern "amdgpu-kernel" fn mu();
}

type TAU = extern "amdgpu-kernel" fn();

The check_abi function…

pub(super) fn check_abi(tcx: TyCtxt<'_>, span: Span, abi: Abi) {
if !tcx.sess.target.is_abi_supported(abi) {
struct_span_err!(
tcx.sess,
span,
E0570,
"The ABI `{}` is not supported for the current target",
abi
)
.emit()
}
// This ABI is only allowed on function pointers
if abi == Abi::CCmseNonSecureCall {
struct_span_err!(
tcx.sess,
span,
E0781,
"the `\"C-cmse-nonsecure-call\"` ABI is only allowed on function pointers."
)
.emit()
}
}

… is not called for the following two ABI lines, meaning that the checks present therein do not activate, potentially allowing circumvention of the checks this function implements. In particular one thing that is allowed is taking the TAU function pointer as an argument and calling it on architectures where this ABI is unsupported.

This becomes especially relevant after #86231 lands.

@nagisa nagisa added the C-bug Category: This is a bug. label Nov 21, 2021
@fmease fmease added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. A-ABI Area: Concerning the application binary interface (ABI) and removed needs-triage-legacy labels Jan 25, 2024
@RalfJung
Copy link
Member

RalfJung commented May 23, 2025

#130260 seems to address this for function pointer types, including in type aliases.

However, trait declarations still seem to be missed:

#![feature(abi_gpu_kernel)]
trait T {
    extern "gpu-kernel" fn mu();
}

Note that attempting to implement such a trait already emits a hard error.

Cc @tdittr -- in case you have an appetite for another lint similar to #128784 :D

@RalfJung RalfJung changed the title The check_abi function in typeck is not called for ABIs in trait declarations or type aliases The check_abi function in typeck is not called for ABIs in trait declarations May 23, 2025
rust-bors bot added a commit that referenced this issue Jun 6, 2025
Reject `extern "{abi}"` when the target does not support it

## What

Promote [`unsupported_fn_ptr_calling_conventions`] from a warning to a hard error, making sure edge-cases will not escape. We now emit hard errors for every case we would return `Invalid` from `AbiMap::canonize_abi` during AST to HIR lowering. In particular, these architecture-specific ABIs now only compile on their architectures[^1]:
  - amdgpu: "gpu-kernel"
  - arm: "aapcs", "C-cmse-nonsecure-entry"
  - avr: "avr-interrupt", "avr-non-blocking-interrupt"
  - msp430: "msp430-interrupt"
  - nvptx64: "gpu-kernel", "ptx-kernel"
  - riscv32 and riscv64: "riscv-interrupt-machine", "riscv-interrupt-supervisor"
  - x86: "thiscall"
  - x86 and x86_64: "x86-interrupt"
  - x86_64: "sysv64", "win64"

The panoply of ABIs that are logically x86-specific but actually permitted on all Windows targets remain supported on Windows, as they were before. For non-Windows targets they error if the architecture does not match.

Moving the check into AST lowering **is itself a breaking change in rare cases**, above and beyond the cases rustc currently warns about. See "Why or Why Not" for details.

## How

We modify rustc_ast_lowering to prevent unsupported ABIs from leaking through the HIR without being checked for target support. Previously ad-hoc checking on various HIR items required making sure we check every HIR item which could contain an `extern "{abi}"` string. This is a losing proposition compared to gating the lowering itself.

As a consequence, unsupported ABI strings will now hard-error instead of triggering the FCW `unsupported_fn_ptr_calling_conventions`.

However, per #86232 this does cause errors for rare usages of `extern "{abi}"` that were theoretically possible to write in Rust source, without previous warning or error. For instance, trait declarations without impls were never checked. These are the exact kinds of leakages that this new approach prevents.

This differs from the following PRs:
- #141435 is orthogonal, as it adds a new lint for ABIs we have not warned on and are not touched by this PR
- #141877 is subsumed by this, in that this simply cuts out bad functionality instead of adding epicycles for stable code

## Why or Why Not

We already made the decision to issue the `unsupported_fn_ptr_calling_conventions` future compatibility warning. It has warned in dependencies since #135767, which reached stable with Rust 1.87. That was released on 2025 May 17, and it is now June. As we already had erred on these ABI strings in most other positions, and warn on stable for function pointer types, this breakage has had reasonable foreshadowing.

Upgrading the warning to an error addresses a real problem. In some cases the Rust compiler can attempt to actually compute the ABI for calling a function. We could accept this case and compute unsupported ABIs according to some other ABI, silently[^0]. However, this obviously exposes Rust to errors in codegen. We cannot lower directly to the "obvious" ABI and then trust code generators like LLVM to reliably error on these cases, either.

Refactoring the compiler so we could defer more ABI computations would be possible, but seems weakly motivated. Even if we succeeded, we would at minimum risk:
- exposing the "whack-a-mole" problem but "approaching linking" instead of "leaving AST"
- making it harder to reason about functions we *can* lower further
- complicating the compiler for no clear benefit

A deprecation cycle for the edge-cases could be implemented first, but it is not very useful for such marginal cases, like this trait declaration without a definition:
```rust
pub trait UsedToSneakBy {
    pub extern "gpu-kernel" fn sneaky();
}
```

Upon any impl, even for provided fn within trait declarations, e.g. `pub extern "gpu-kernel" fn sneaky() {}`, different HIR types were used which would, in fact, get checked. Likewise with anything with function pointers. Thus we would be discussing deprecation cycles for code that is impotent or forewarned[^2].

Implementing a deprecation cycle _is_ possible, but it would likely require emitting multiple of a functionally identical warning or error on code that would not have multiple warnings or errors before. It is also not clear to me we would not find **another**, even more marginal edge-case that slipped through, as "things slip through" is the motivation for checking earlier. Additional effort spent on additional warnings should require committing to a hard limit first.

r? lang

Fixes #86232
Fixes #132430
Fixes #138738
Fixes #142107

[`unsupported_fn_ptr_calling_conventions`]: #130260
[^1]: Some already will not compile, due to reaching ICEs or LLVM errors.
[^0]:  We already do this for all `AbiStr` we cannot parse, pretending they are `ExternAbi::Rust`, but we also emit an error to prevent reaching too far into codegen.
[^2]: It actually did appear in two cases in rustc's test suite because we are a collection of Rust edge-cases by the simple fact that we don't care if the code actually runs. These cases were excised in c1db989.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-ABI Area: Concerning the application binary interface (ABI) C-bug Category: This is a bug. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants