Incus 6.11 breaks Nvidia Compute Runtime

[lqvnguyen]

After updating to the latest version of Incus 6.11, I can no longer start any of my containers that have the following in its config or part of a profile:

config:
nvidia.driver.capabilities: all
nvidia.require.cuda: "true"
nvidia.runtime: "true"

incus start ai
Error: Failed to run: /opt/incus/bin/incusd forkstart ai /var/lib/incus/containers /run/incus/ai/lxc.conf: exit status 1
Try incus info --show-log ai for more info

incus info --show-log ai
Name: ai
Description:
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2025/02/22 02:19 PST
Last Used: 2025/04/04 19:08 PDT

Log:

lxc ai 20250405020851.656 ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1
lxc ai 20250405020851.656 ERROR conf - ../src/lxc/conf.c:lxc_setup:3948 - Failed to run mount hooks
lxc ai 20250405020851.656 ERROR start - ../src/lxc/start.c:do_start:1268 - Failed to setup container "ai"
lxc ai 20250405020851.656 ERROR sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc ai 20250405020851.662 WARN network - ../src/lxc/network.c:lxc_delete_network_priv:3674 - Failed to rename interface with index 0 from "eth0" to its initial name "veth129a886f"
lxc ai 20250405020851.662 ERROR start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "ai"
lxc ai 20250405020851.662 WARN start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 6716
lxc ai 20250405020851.662 ERROR lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc 20250405020851.750 ERROR af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20250405020851.750 ERROR commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

[ChuckPa]

I also have this error.

https://discuss.linuxcontainers.org/t/incus-6-11-update-containers-with-nvidia-gpu-passthrough-will-not-start/23400

[stgraber]

Looking into this one now. I think I know what the problem may be.

[lqvnguyen]

Thank you. This is disrupting Plex QA

[ChuckPa]

Thank you very much

[lqvnguyen]

@stgraber I tried to roll back to 6.0.4 LTS and encountered the same issue

[lqvnguyen]

@stgraber Is there any workarounds with the configuration?

[stgraber]

Not that I'm aware of, other than turning nvidia.runtime to false.

[stgraber]

@mihalicyn I think the dumpable change may have caused this.

I've confirmed that reverting to 6.10.1 doesn't seem to help and we didn't bump nvidia-container recently.
There is a bug in Incus 6.11 that causes part of this issue but even with a fix in place, the hook still fails for me during a setns call.

[stgraber]

lxc u24 20250405044621.838 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /opt/incus/share/lxc/hooks/nvidia produced output: mkdir: cannot create directory ‘/var/lib/incus/storage-pools/default/containers/u24/hook’: Permission denied

lxc u24 20250405044621.841 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /opt/incus/share/lxc/hooks/nvidia produced output: + exec nvidia-container-cli --user configure --no-cgroups --ldconfig=@/usr/sbin/ldconfig.real --compute --utility --require= --require= /opt/incus/lib/lxc/rootfs

lxc u24 20250405044621.876 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /opt/incus/share/lxc/hooks/nvidia produced output: nvidia-container-cli: mount error: open failed: /opt/incus/lib/lxc/rootfs/proc/1/ns/mnt: permission denied

lxc u24 20250405044621.879 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1

[stgraber]

@mihalicyn lxc/lxc#4536 may do the trick. Going to push that to our daily packages and re-check with both the OCI and nvidia cases.

[stgraber]

Right, so the LXC fix above combined with lxc/incus#1895 fixes it here

[lqvnguyen]

Will we see new versions of LTS & Stable soon?

[stgraber]

I'll be updating the Zabbly repository with cherry picked version of the two needed fixes later today.

[ChuckPa]

I downloaded and installed updates from the repo.

I now have working HW passthrough again.

root@plex:~# nvidia-smi
Sat Apr  5 23:28:28 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.77                 Driver Version: 565.77         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX 2000 Ada Gene...    Off |   00000000:01:00.0  On |                  Off |
| 30%   54C    P2             47W /   70W |    2407MiB /  16380MiB |     37%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      5727      C   ...lib/plexmediaserver/Plex Transcoder        522MiB |
+-----------------------------------------------------------------------------------------+
root@plex:~# 

Thank you for jumping on and solving this quickly.