GitHub workflow: set RPATH to "@loader_path" / "$ORIGIN" to ensure executables and dynamic libraries search for dependencies in their origin directory.

[rotemdan]

(Some earlier information in related pull request: #13741)

Linux binaries

Currently the distributed Linux binaries (executables and shared libraries) default to embed an RPATH with an absolute path only relevant to the CI runner, like /home/runner/work/llama.cpp/llama.cpp/build/bin:

readelf -d llama-cli

Dynamic section at offset 0x1bcd80 contains 36 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libllama.so]
 0x0000000000000001 (NEEDED)             Shared library: [libggml.so]
 0x0000000000000001 (NEEDED)             Shared library: [libggml-base.so]
 0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000001d (RUNPATH)            Library runpath: [/home/runner/work/llama.cpp/llama.cpp/build/bin:]

This means that the binaries can only run correctly if in the current working directory, or if on system path.

If trying to run not from the current working directory, an error occurs:

llama-bin/llama-cli

llama-bin/llama-cli: error while loading shared libraries: libggml.so: 
cannot open shared object file: No such file or directory

After adding:

            -DCMAKE_INSTALL_RPATH="$ORIGIN" \
            -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \

Linux binaries should include the $ORIGIN RPATH instead:

readelf -d llama-cli

Dynamic section at offset 0x196e68 contains 34 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libllama.so]
 0x0000000000000001 (NEEDED)             Shared library: [libggml.so]
 0x0000000000000001 (NEEDED)             Shared library: [libggml-base.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN]

Which resolves this issue.

Issues with Linux shared libraries

Similar issue occurs with the built shared libraries. It's difficult to use them since their search paths don't allow finding each other in the same origin directory. This resolves that as well.

macOS binaries

For macOS, the workflow already had a related build setting:

            -DCMAKE_BUILD_RPATH="@loader_path" \

Which in practice did something similar, since no CMAKE install phase was likely used.

otool -l ./llama-cli | grep LC_RPATH -A2
          cmd LC_RPATH
      cmdsize 32
         path @loader_path (offset 12)
--
          cmd LC_RPATH
      cmdsize 64
         path /Users/runner/work/llama.cpp/llama.cpp/build/bin (offset 12)

It caused both the @loader_path and the runner absolute path to be used.

I also changed to use the same approach:

            -DCMAKE_INSTALL_RPATH="@loader_path" \
            -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \

Which shouldn't actually make any practical difference, except removing the absolute path.

I can't test the workflows

I hope these changes are correct, but I can't exactly test the workflows before submitting the pull request.

Order of command line arguments

I've put the new arguments at the top to be consistent with the order they were added in the macOS builds (in my personal builds I put them last actually). You can change the order in any way you want.

[slaren]

You can test this by pushing to master in your fork.

[rotemdan]

Thanks.

It took a long time, but the 4 relevant builds did run, and some of the Windows builds failed due to build environment issues (Error: could not load cache). The changes made don't impact the Windows builds at all, so it's very unlikely to be related (errors happened during initialization):

https://github.com/rotemdan/llama.cpp/actions/runs/15822554770

Second run (up-to-date from upstream), got same failures due to the build environment:

https://github.com/rotemdan/llama.cpp/actions/runs/15823143822

When I downloaded the resulting Linux binaries and ran readelf on them none of them had any RPATH set at all. Running them gave errors they couldn't find the .so files, even in the local working directory.

I don't know what can cause it, the parameters seem correct, because I've used them many times myself, locally, both in llama.cpp and whisper.cpp.

It could be something about the build environment in the fork isn't set up correctly, otherwise the Windows builds should have succeeded, since I've made no changes to them.

(I've already spent more than 1/2 of the day today working exclusively on various llama.cpp issues. These are all very technical issues that are difficult to diagnose and deal with. At the moment it doesn't seem like I'm able to recreate the exact build environment that llama.cpp repository is using in its workflow runner, in the fork. It could be a simple setting, but the amount of effort required to figure it out may be large. I've never used GitHub workflows before.)

[slaren]

My guess is that the $ORIGIN is being interpreted by the shell as an environment variable, so you are actually running cmake with DCMAKE_INSTALL_RPATH="".

[rotemdan]

Based on a suggestion from Gemini, I changed to use single quotes ('$ORIGIN', '@loader_path') instead of double quotes and now it seems to work as expected on the Linux build.

Actually, I locally always used single quotes. The only reason I changed to double quotes was because I wanted to be consistent the original workflow script. I wasn't sure if there was a difference between the two. It turns out there was.

The updated run is here (I rebased with the latest changes, so recent upstream changes to release.yml are included):

https://github.com/rotemdan/llama.cpp/actions/runs/15827690592

I'll test the macOS automated builds next. Already downloaded them.

[rotemdan]

I can't really run the macOS build, because I'm using macOS 13.0 over VMWare, which apparently is too old for them.

I did verify the RPATH is set up correctly in the executables and .dylib files.

Anyway, here are the links to the build artifacts .zip files:

ubuntu-22-cpu (x64, ubuntu-22.04):

• https://github.com/rotemdan/llama.cpp/actions/runs/15827690592/artifacts/3384118181

ubuntu-22-vulkan:

• https://github.com/rotemdan/llama.cpp/actions/runs/15827690592/artifacts/3384092165

macOS-x64:

• https://github.com/rotemdan/llama.cpp/actions/runs/15827690592/artifacts/3384079517

macOS-arm64:

• https://github.com/rotemdan/llama.cpp/actions/runs/15827690592/artifacts/3384076103