cmake : set RPATH to $ORIGIN on Linux (#13740)

[sunhaitao]

This patch makes the commands callable from any working directory.

[rotemdan]

This is crucial for enabling the official distribution to be used as a true portable dynamically linked library. Currently the embedded RPATH points to an absolute path that almost no one would have (/home/runner/work/llama.cpp/llama.cpp/build/bin):

readelf -d libllama.so

Dynamic section at offset 0x1bb5a0 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libggml.so]
 0x0000000000000001 (NEEDED)             Shared library: [libggml-base.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000e (SONAME)             Library soname: [libllama.so]
 0x000000000000001d (RUNPATH)            Library runpath: [/home/runner/work/llama.cpp/llama.cpp/build/bin:]

So basically without setting a global LD_LIBRARY_PATH, the .so files are mostly unusable in the real-world. They are part of the distribution, but at their current state aren't really ready to be used in an effective way. Setting a global LD_LIBRARY_PATH is cumbersome and just unnecessary. It kind of defeats the purpose of dynamic linking via, say, dlopen.

Why it's a significant problem

It's a significant issue for a wrapper library I'm publishing, since at the current state, users can't use the official distribution .so files because they simply won't load without a global LD_LIBRARY_PATH set, which must be set before the process starts (if updated at runtime, changes are ignored), which makes it unusable for a library.

Adding similar arguments for macOS builds

Same can be done for macOS builds.

When I compile I add:

-D CMAKE_INSTALL_RPATH='@loader_path' -D CMAKE_BUILD_WITH_INSTALL_RPATH=ON

[slaren]

You can run cmake --install to get the binaries with the correct RPATH.

[rotemdan]

Default cmake --install will just put an absolute RPATH to the local directory where the compilation occurred. The resulting .so files will not be usable for a portable distribution, other than the machine where they were compiled.

To fix that, if I build myself, I use -D CMAKE_INSTALL_RPATH='$ORIGIN' -D CMAKE_BUILD_WITH_INSTALL_RPATH=ON. This means that the transitive dependencies of libllama.so.1 would be searched in its local directory.

My own cmake command looks like this:

rm -r build

cmake -B build -D GGML_CUDA=OFF -D BUILD_SHARED_LIBS=ON \
-D LLAMA_BUILD_COMMON=OFF -D LLAMA_BUILD_TESTS=OFF \
-D LLAMA_BUILD_TOOLS=OFF -D LLAMA_BUILD_EXAMPLES=OFF \
-D LLAMA_BUILD_SERVER=OFF -D LLAMA_CURL=OFF \
-D CMAKE_INSTALL_RPATH='$ORIGIN' -D CMAKE_BUILD_WITH_INSTALL_RPATH=ON

cmake --build build --config Release -j 4

Of course I can rebuild myself, but I can't do that in the rapid pace that the library gets updated at, or have access to all platforms.. It's much more productive to use the official dynamic libraries from the official distribution.

The .so files in the official distribution, on GitHub releases can't be used for distribution as part of usable production libraries, because their RPATH is an absolute directory that is no applicable to anywhere else other than the CI runner, so transitive dependencies can only be resolved correctly if a global LD_LIBRARY_PATH path is set (before the running process starts - cannot be while it's running), but that is not a requirement that is really necessary, or usable and a lot of time is not possible at all. It's a kind of last resort hack.

Edit

I gave the whisper.cpp compilation command line by mistake! Sorry. Fixed it! I work with both libraries.

[slaren]

The linux releases are very bare bones, I cannot recommend using them for anything other than testing. However, it would be ok to modify the way the linux releases are built in release.yml so that the binaries have the correct RPATH, rather than modifying the cmake script and have it apply by default.

[rotemdan]

It doesn't personally matter to me if the default build parameters set RPATH to search in the local directory, since if I do build myself, I use custom build parameters anyway (which took a lot of time and LLM assistance to refine, though).

Having the official released binaries have RPATH settings that are more usable towards portable distributions would be great, since they are already effectively distributed in a "portable" way, implying you could just copy and use them.

Also, in the releases, the executables themselves, like llama-cli embed an absolute RPATH that is only relevant for the CI runner:

readelf -d llama-cli

Dynamic section at offset 0x1bbd80 contains 36 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libllama.so]
 0x0000000000000001 (NEEDED)             Shared library: [libggml.so]
 0x0000000000000001 (NEEDED)             Shared library: [libggml-base.so]
 0x0000000000000001 (NEEDED)             Shared library: [libcurl.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000001d (RUNPATH)            Library runpath: [/home/runner/work/llama.cpp/llama.cpp/build/bin:]

If you run it, it still works, because likely that on Linux there is a fallback to search dependencies at the current working directory. However, in slightly more complex applications, like mine, I cannot guarantee what the CWD would be, and it's not a good idea to modify it, even temporarily (unsafe, especially threading involved!). The simple and direct way to resolve this is likely ensure that the binary files are designed to directly "find each other" in the same directory they are located. It assumes a portable distribution which is pretty much how the releases are organized.

Also, the change should probably also be done for the macOS .dylib and binaries as well (using @loader_path instead of $ORIGIN).

I could try to give a pull request but I noticed there are many different files involved in the CI, with some custom shell scripts, so I don't really know where should I make the changes. It's a bit overwhelming.

[slaren]

I could try to give a pull request but I noticed there are many different files involved in the CI, with some custom shell scripts, so I don't really know where should I make the changes. It's a bit overwhelming.

The linux releases are created entirely here, the other files are not relevant.

llama.cpp/.github/workflows/release.yml

Lines 128 to 235 in 06cbedf

	ubuntu-22-cpu:
	strategy:
	matrix:
	include:
	- build: 'x64'
	os: ubuntu-22.04
	# GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS are not currently supported on arm
	# - build: 'arm64'
	# os: ubuntu-22.04-arm
	runs-on: ${{ matrix.os }}
	steps:
	- name: Clone
	id: checkout
	uses: actions/checkout@v4
	with:
	fetch-depth: 0
	- name: ccache
	uses: hendrikmuhs/[email protected]
	with:
	key: ubuntu-cpu-cmake
	evict-old-files: 1d
	- name: Dependencies
	id: depends
	run: |
	sudo apt-get update
	sudo apt-get install build-essential libcurl4-openssl-dev
	- name: Build
	id: cmake_build
	run: |
	cmake -B build \
	-DGGML_BACKEND_DL=ON \
	-DGGML_NATIVE=OFF \
	-DGGML_CPU_ALL_VARIANTS=ON \
	-DLLAMA_FATAL_WARNINGS=ON \
	${{ env.CMAKE_ARGS }}
	cmake --build build --config Release -j $(nproc)
	- name: Determine tag name
	id: tag
	uses: ./.github/actions/get-tag-name
	- name: Pack artifacts
	id: pack_artifacts
	run: |
	cp LICENSE ./build/bin/
	zip -r llama-${{ steps.tag.outputs.name }}-bin-ubuntu-${{ matrix.build }}.zip ./build/bin/*
	- name: Upload artifacts
	uses: actions/upload-artifact@v4
	with:
	path: llama-${{ steps.tag.outputs.name }}-bin-ubuntu-${{ matrix.build }}.zip
	name: llama-bin-ubuntu-${{ matrix.build }}.zip
	ubuntu-22-vulkan:
	runs-on: ubuntu-22.04
	steps:
	- name: Clone
	id: checkout
	uses: actions/checkout@v4
	with:
	fetch-depth: 0
	- name: ccache
	uses: hendrikmuhs/[email protected]
	with:
	key: ubuntu-22-cmake-vulkan
	evict-old-files: 1d
	- name: Dependencies
	id: depends
	run: |
	wget -qO - https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo apt-key add -
	sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list
	sudo apt-get update -y
	sudo apt-get install -y build-essential mesa-vulkan-drivers vulkan-sdk libcurl4-openssl-dev
	- name: Build
	id: cmake_build
	run: |
	cmake -B build \
	-DGGML_BACKEND_DL=ON \
	-DGGML_NATIVE=OFF \
	-DGGML_CPU_ALL_VARIANTS=ON \
	-DGGML_VULKAN=ON \
	${{ env.CMAKE_ARGS }}
	cmake --build build --config Release -j $(nproc)
	- name: Determine tag name
	id: tag
	uses: ./.github/actions/get-tag-name
	- name: Pack artifacts
	id: pack_artifacts
	run: |
	cp LICENSE ./build/bin/
	zip -r llama-${{ steps.tag.outputs.name }}-bin-ubuntu-vulkan-x64.zip ./build/bin/*
	- name: Upload artifacts
	uses: actions/upload-artifact@v4
	with:
	path: llama-${{ steps.tag.outputs.name }}-bin-ubuntu-vulkan-x64.zip
	name: llama-bin-ubuntu-vulkan-x64.zip

[rotemdan]

Update: this may actually be seen as a bug

If you try to run llama-cli from a location other than its own directory (CWD is different than the one containing the .so files), you get an error:

llama-bin/llama-cli: error while loading shared libraries: libggml.so: cannot open shared object file: No such file or directory

The suggested change is expected to fix this (edit: need to verify to be 100% sure).

[rotemdan]

I tested changing to $ORIGIN allows running the executables from other directories (making them more like a true portable app).

I created pull request #14309 with the changes (macOS builds already had related CMAKE arguments, it turns out).

[sunhaitao]

Since many people need to build for themselves on Linux, I'm afraid that simply changing release.yml is not helpful enough.

[slaren]

The way I see it, there are essentially two reason to build llama.cpp on Linux:

• You are a developer and are using the binaries from the build directory, in which case you want the RPATH that cmake sets to ensure that you are using the libraries that you just built
• You are an user and want to install llama.cpp, in which case you will run cmake --install afterwards and the RPATH will be set correctly

[rotemdan]

Here's how I see it:

The build process puts all the binaries in a single flat directory, which makes things easy for distribution and deployment.

So, if the user is interested in copying those to another location, it kind of seems like everything should work just fine, but in practice, without special compiler parameters (which are kind of obscure and not known to most users), it wouldn't really work as expected due to the absolute search paths the binaries embed (unless binaries are in global path or in current working directory).

The absolute search paths, at this stage, are not "installation" paths. They are local paths to the build directory used for compilation. They don't actually provide a real "service" to the user.

Actually they could be a source of confusion and error, since if the binary is copied to another location, and the binary embeds an absolute search path to a build directory (which is highest priority in search), later the build directory may contain different, incompatible builds, so the copied binary might work initially, but then mysteriously fail because the shared library files in the build directory has been recompiled to a different, incompatible version.

[sunhaitao]

* You are a developer and are using the binaries from the `build` directory, in which case you want the RPATH that cmake sets to ensure that you are using the libraries that you just built

Setting RPATH to $ORIGIN ensures that shared libraries in the same build tree, which are the libraries just built, are used.

* You are an user and want to install llama.cpp, in which case you will run `cmake --install` afterwards and the RPATH will be set correctly

However, this approach generates binaries that cannot be moved to another path. Setting RPATH to $ORIGIN ensures the binaries remain functional even after being moved, which can save users significant trouble in situations like saving an old working version. And for users who choose to run cmake --install, setting RPATH to $ORIGIN by default does no harm.

There is also a specific scenario where setting RPATH to $ORIGIN is particularly helpful: you are a developer building a product that bundles llama.cpp as a runtime. In such cases, neither hardcoding a specific location nor calling cmake --install is practical.

Since in all three of these situations, setting RPATH to $ORIGIN brings some benefits and sacrifices nothing, I believe it is a good Pareto improvement to adopt.