`cargo run` exports bad DYLD_LIBRARY_PATH on macOS

[casey]

I've been going down the rabbit hole for a while on this one. Thankfully though, it seems like it's actually hopefully a pretty simple fix. Or, if it isn't, at least maybe I'm reporting the issue in the right repo ^_^

I originally thought it was a git2-rs issue, and reported it in https://github.com/alexcrichton/git2-rs/issues/175. Then I thought it was a rustc issue, and reported it here: rust-lang/rust#36250.

Finally though, I've gotten to the bottom of it, and it appears to be a cargo bug.

A few things need to happen to trigger the issue:

1. We are linking ina dynamic library that is not provided by the system, for example something from homebrew in /usr/local/lib, or from macports in /opt/local/lib.

2. We are also linking in a system framework.

3. In the nonstandard location in 1., there is a dynamically linked library which the system framework from 2. needs. However, this library in 1. is not the version that the framework expects.

4. We run the binary with cargo run.

To make everything concrete, let's say that we're linking in openssl provided by macports in /opt/local/lib, the framework we're using is the ApplicationServices framework, and we also happen to have libiconv installed in /opt/local/lib.

Cargo will export DYLD_LIBRARY_PATH=/opt/local/lib:..., and we'll get the following error:

: cargo run
   Compiling rust-dynamic-linker-error v0.0.0 (file:///Users/rodarmor/src/rust-dynamic-linker-error)
    Finished debug [unoptimized + debuginfo] target(s) in 0.34 secs
     Running `target/debug/rust-dynamic-linker-error`
dyld: Symbol not found: _iconv
  Referenced from: /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
  Expected in: /opt/local/lib/libiconv.2.dylib
 in /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling

What's happening is that the LanguageModeling framework, which is a transitive dependency of the ApplicationServices framework also depends on libiconv. Normally it finds it in /usr/lib/libiconv.dylib. However, because of the value of DYLD_LIBRARY_PATH, it finds it in /usr/local/lib/libiconv.dylib, and we get the error because it is the wrong version with a missing symbol.

If we run the binary directly, without the environment variable, it works:

: ./target/debug/rust-dynamic-linker-error
Hello, world!

Although the conditions needed to trigger this bug may seem esoteric, they are actually very common on mac development machines. Developers often have a ton of stuff installed via macports or homebrew.

There are a few ways that this might be resolved:

The DYLD_LIBRARY_PATH value seems to be unnecessary at least in this case, since the binary works when run directly. If so, perhaps we can avoid exporting it so as not to cause this breakage? However, perhaps there are other situations where the variable is necessary to find libraries? What's the purpose of the export?

If the DYLD_LIBRARY_PATH value is necessary, then we need to communicate that to developers somehow. It would be very frustrating to have a binary which runs with cargo run, but which cannot be run directly.

And, if we do need DYLD_LIBRARY_PATH, then we should consider if DYLD_FALLBACK_LIBRARY_PATH might suffice. DYLD_FALLBACK_LIBRARY_PATH does not disrupt the normal order of dynamic library lookup, but instead is only consulted if the dynamic linker cannot find a library in the normal places.

[alexcrichton]

This is a pretty tricky issue but I think the solution is that we should probably just not try to add any "system directories" to DYLD_LIBRARY_PATH. We should basically just assume that any directory outside the build directory is up to the user to be in DYLD_LIBRARY_PATH as it's not generated by Cargo itself.

@casey would you be willing to help test out a patch for this? I think the relevant section of code is here and we likely just want to filter out paths that aren't in the build directory.

[casey]

Sure, I'd be happy to.

One thing I'm curious about, when is DYLD_LIBRARY_PATH necessary for a binary to work? All my binaries seem to run fine without it.

[alexcrichton]

It's sometimes needed for plugins but it's primarily used for build scripts which generate dynamic native libraries.

[pkgw]

I've run into the same problem with cargo test (example) and am happy to test, or to try to come up with a patch that implements the logic to filter out paths outside of the build directory.

[alexcrichton]

@pkgw ok! If you need any help just let me know.

I think the basic patch here is to just only append paths that look like they're in the cargo build directory, and ignore others.

[pkgw]

OK, I've create a pull request, #3651, that applies this strategy. I was even able to test it and it works in my test case!

However, I have to admit that I mostly forget what's even going on in this code, so I'm far from confident that it's doing the right thing. Fortunately it's only a few lines' worth of changes.

My test case:

1. Get a Mac with Homebrew
2. Install at least libjpeg and libpng using Homebrew.
3. Create a Cargo project with contents as follows.
4. For Cargo.toml:

[package]
name = "issue3366"
version = "0.0.1"
authors = ["Peter Williams <peter@newton.cx>"]
build = "build.rs"

[lib]
name = "mytest"
crate-type = ["rlib"]

[build-dependencies]
gcc = "^0.3"

5. For build.rs:

extern crate gcc;

fn main() {
  let mut cfg = gcc::Config::new();
  cfg.file("blah.c");
  println!("cargo:rustc-link-lib=framework=ImageIO");
  println!("cargo:rustc-link-lib=dylib=jpeg");
  println!("cargo:rustc-link-search=/usr/local/lib");
  cfg.compile("libfoo.a");
}

6. For blah.c:

int myfunction (void) { return 3; } 

7. For src/lib.rs:

pub fn hello() -> usize { 3 }

8. Run cargo test. With the buggy version, I get a dyld: Symbol not found __cg_jpeg_resync_to_restart error. With the patched version, it just works.

However, I just ran the test suite and the patch definitely breaks the run_with_library_paths test. If the discussion in this issue is correct, the test should just be deleted, I think.

[sgrif]

Oh hey, I was updating the README for pq-sys to have more information about this and noticed I couldn't reproduce the issue anymore. Just wanted to confirm that #3651 did in fact fix a long standing issue for us.

[sgrif]

I do agree with the assessment that setting DYLD_FALLBACK_LIBRARY_PATH is the right thing to do. Here's another case that can trigger the problem on macOS. Many people on Mac install PostgreSQL using Postgres.app. That program places libpq.dylib in a subdirectory of the executable, and also vendors libjpeg.dylib. If we link against libpq when someone has installed it using that method, that directory will be added to DYLD_LIBRARY_PATH, and the same issue will occur with dyld: Symbol not found __cg_jpeg_resync_to_restart

[alexcrichton]

@sgrif thanks for confirming the fix! I'd also be totally up for leveraging DYLD_FALLBACK_LIBRARY_PATH. It's unfortunate that it's OSX-specific, but hey if it fixes a platform it fixes a platform.