`rustc` proxy may fail to invoke a `rustc` using shared libraries even when `LD_LIBRARY_PATH` is set.

[cr1901]

Problem
I have a custom Rust toolchain for development installed locally, call it rust-shared. I run it by setting LD_LIBRARY_PATH whenever I need to e.g. invoke the rustup cargo proxy. In the past, rustup has been able to call my custom Rust toolchain just fine, including locating its shared libraries. However, as of a recent release (I don't know which), I've found that rustup is inconsistent as to whether invoking my custom rustc will succeed or fail to find its shared libraries even when LD_LIBRARY_PATH has been set previously.

This behavior is inconsistent, but I've found a way to trigger the problem consistently via a contrived sequence.

Steps
Prerequisites

1. Compile a rustc using a shared LLVM :).
2. Add this toolchain to rustup via rustup toolchain link shared-rust /path/to/rust/root
3. Create a sample crate, e.g. cargo init --lib testing
4. Compile the crate using your shared toolchain: LD_LIBRARY_PATH=/path/to/llvm/libs cargo +shared-rust build. This should succeed.

rustup Seems To Find Shared Libraries Without LD_LIBRARY_PATH
Up to this point, all commands have expected behavior for me. Steps 5 and 6 have odd behavior to me.

5. Run cargo +shared-rust clean once without setting LD_LIBRARY_PATH. This succeeds on my machine. I would not expect this to succeed because I didn't specify where the shared libraries for rustc live. I've accidentally forgotten to set LD_LIBRARY_PATH a number of times in normal usage :).
6. Run cargo +shared-rust clean again without setting LD_LIBRARY_PATH. This fails with "error while loading shared libraries", which is the behavior I expected in step 5.

rustup Seems To Disregard LD_LIBRARY_PATH
If I rebuild the crate using +shared-rust at this point, compilation will still succeed. However, the next set of commands will result in shared-rust not finding its shared libraries.

7. Run cargo +nightly build. This should succeed just fine.

8. Run cargo +shared-rust clean once without setting LD_LIBRARY_PATH. This fails with "error while loading shared libraries" on my machine, as I would expect.

9. We've already shown in step 4 that compilation with shared-rust works. However, when I attempt to run LD_LIBRARY_PATH=/path/to/llvm/libs cargo +shared-rust build or LD_LIBRARY_PATH=/path/to/llvm/libs cargo +shared-rust clean after running steps 7 and 8, I get the following error:

   william@xubuntu-dtrain:~/Projects/toolchains/testing$ LD_LIBRARY_PATH=$SHARED_LLVM_LIBS cargo +shared-rust build
   error: process didn't exit successfully: `rustc -vV` (exit status: 127)
   --- stderr
   /home/william/.rustup/toolchains/shared-rust/bin/rustc: error while loading shared libraries: libLLVMX86AsmParser.so.12: cannot open shared object file: No such file or directory
   

   Notice that I explicitly set LD_LIBRARY_PATH, yet rustc still can't find its shared libraries!

10. Running cargo +nightly clean or cargo +nightly build makes compilation with shared-rust functional again, like in step 4.

Possible Solution(s)
I'm personally unsure how to debug this further. It seems that cargo +toolchain clean is the trigger to switch rustup's state between "will find the shared libraries" versus "will not find the shared libraries". I think the inconsistent behavior I saw in step 5 and 6 is related, as its perhaps "toggling" some internal state in rustup?

Running LD_LIBRARY_PATH=/path/to/llvm/libs cargo +shared-rust clean does not trigger the above behavior in steps 5 and beyond, unless we're already in the "rustc has decided not to find its shared libraries" state, like in step 9.

Notes

System Information:

william@xubuntu-dtrain:~/Projects/toolchains/rust$ uname -a
Linux xubuntu-dtrain 5.4.0-72-generic rust-lang/rustup#80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

william@xubuntu-dtrain:~/Projects/toolchains/rust$ rustup -V
rustup 1.24.1 (a01bd6b0d 2021-04-27)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.53.0-nightly (9d9c2c92b 2021-04-19)`

william@xubuntu-dtrain:~/Projects/toolchains/rust$ cargo +nightly -Vv
cargo 1.53.0-nightly (65d57e6f3 2021-04-04)
release: 1.53.0
commit-hash: 65d57e6f384c2317f76626eac116f683e2b63665
commit-date: 2021-04-04

william@xubuntu-dtrain:~/Projects/toolchains/rust$ rustc +nightly -Vv
rustc 1.53.0-nightly (9d9c2c92b 2021-04-19)
binary: rustc
commit-hash: 9d9c2c92b834c430f102ea96f65119e37320776e
commit-date: 2021-04-19
host: x86_64-unknown-linux-gnu
release: 1.53.0-nightly
LLVM version: 12.0.0

william@xubuntu-dtrain:~/Projects/toolchains/rust$ LD_LIBRARY_PATH=$SHARED_LLVM_LIBS rustc +shared-rust -Vv
rustc 1.54.0-dev
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 1.54.0-dev
LLVM version: 12.0.0

shared-rust is rustc 1.54.0-dev, commit e5f83d24aee. I can provide cmake invocations for LLVM and the config.toml I used to build upon request.

[ehuss]

I can try to clarify what is happening.

Cargo keeps a cache of some rustc information such as its version stored in the target directory (.rustc_info.json).

In step 5, cargo loaded the rustc info from the cache, and avoided running rustc and deleted the target directory.

In step 6, since the cache is gone, it tried to run rustc and failed.

In step 8, the cache was updated that rustc fails to run.

In step 9, cargo (incorrectly) reuses the cache without re-running rustc and just redisplays the error (thinking nothing has changed).

I think, ultimately, this is a cargo issue, where it is incorrectly reusing the cache. If you would like to move this issue over to https://github.com/rust-lang/cargo/issues/, we can follow up from there.

[kinnison]

@rust-lang/infra could one of you move this issue as per @ehuss 's comment above?

[correabuscar]

tl:dr: To repro just chmod a-x your/rustc, cargo clean then cargo build, now it's cached, do chmod u+x your/rustc, run cargo build, same (cached)error.

---

This cache reusal in step 9, I'm seeing it happen for my rustc, if I run it too quickly, without doing a cargo clean first, it caches that this Permission denied happened because not all dirs have execute attribute yet:

error: process didn't exit successfully: `/home/user/sandbox/rust/05_sandbox/tests/exit1/myrust -vV` (exit status: 126)
--- stderr
/home/user/sandbox/rust/05_sandbox/tests/exit1/myrust: line 10: /var/tmp/portage/dev-lang/rust-1.76.0-r1/work/rustc-1.76.0-src/build/x86_64-unknown-linux-gnu/stage1/bin/rustc: Permission denied

but then later when the dirs have all the right permissions, it re-uses that cache and keeps displaying it, so I've to either cargo clean or delete target/.rustc_info.json then it picks up the changes.