Pointer-to-pointer + casts in FFI leads to hard-to-debug mysterious problems

[huonw]

If you have a C API taking a pointer-to-a-pointer, like void**, where the function fills in a pointer value, it's very easy to screw up if there is any pointer casting involved:

extern crate libc;
extern {
    fn c_func(x: *mut *mut libc::c_void);
}

fn main() {
    let x = 0 as *mut u8;
    c_func(&mut (x as *mut libc::c_void));
    println!("new pointer is {}", x);
}

This will always print new pointer is 0x0, no matter what c_func does.

Reason: the x as *mut ... cast is creating a temporary, that's disconnected from the original x and thus the modification happens to the anonymous stack slot that stores the result of the cast. The code should be written something like (&mut x) as *mut _ as *mut *mut libc::c_void.

This is really subtle to debug, so we could have a lint that assists in this case: "did you mean to take a reference to the result of a cast in this FFI call" (could have it apply to non-FFI things too, and presumably it should only apply when there are &mut pointers involved).

[steveklabnik]

Triage: i've created a repository to assist anyone who wants to write this lint: https://github.com/steveklabnik/rust-issue-17417

[steveklabnik]

Since new lints have a big impact on users of rustc, the policy is that they should go through the RFC process like other user-facing changes. As such, I'm going to give this one a close, but if anyone comes across this ticket and wants this lint, consider adding it to clippy and/or writing up an RFC. Thanks!

[Envek]

@huonw, thank you for sharing a recipe for working with pointer-to-a-pointer APIs. It apparently should be documented in FFI chapter of the Rust book!

[joshhansen]

Yes, that formula really does belong in the FFI section of the book. Saved me a million years of experimentation.

[messense]

Just ran into this today, it really should be documented in the book!

[woodgear]

is there some update?