incus file push fails with latest 6.0

[jdstrand]

Hi!

I use the zabbly 6.0-stable packages on an Ubuntu 20.04 amd64 host (an incus vm) with 5.4.0-205-generic kernel (5.4.0-205.225-generic) and have a cron job that does this on the host:

incus file push /path/to/file container/path/to/dir/

The cron job runs as root (but the problem remains when running manually as non-root), the source is root owned, the target is an Ununtu 18.04 container and the target dir is owned by root within the container. The container uses a dedicated uid range (idmap.isolated is true, idmap.size is 1000000, idmap.base 7000000).

After upgrading to 1:6.0.3-ubuntu20.04-202501311435 from 1:6.0.3-ubuntu20.04-202412201836, this command fails with:

Error: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF

and I see this in the logs:

Feb 15 13:10:01 srchost incusd[1107]: time="2025-02-15T13:10:01-06:00" level=error msg="SFTP server stopped with error" err="exit status 1" instance=dstcontainer instanceType=container project=default stderr="error: setns: Invalid argument\nerrno: 22"
Feb 15 13:10:01 srchost incusd[1107]: time="2025-02-15T13:10:01-06:00" level=warning msg="Failed copying SFTP instance connection to remote connection" err="read unix @->/proc/self/fd/23/forkfile.sock: read: connection reset by peer" instance=dstcontainer local=/var/lib/incus/unix.socket project=default remote=@

Downgrading back to 1:6.0.3-ubuntu20.04-202412201836 resolves the issue.

I saw this issue that may be related but seems different: #1614

[stgraber]

Can you show the full incus config show --expanded dstcontainer?
That should make it easier to reproduce.

The issue is likely with the namespace attaching logic that's been cherry-picked into the package and which would also mean that 6.8 and 6.9 should be similarly affected.

[jdstrand]

Sure, here you go (with downgraded 1:6.0.3-ubuntu20.04-202412201836):

architecture: x86_64
config:
  boot.autostart: "true"
  linux.kernel_modules: br_netfilter
  security.idmap.base: "7000000"
  security.idmap.isolated: "true"
  security.idmap.size: "1000000"
  volatile.eth0.host_name: veth9e020dcf
  volatile.eth0.hwaddr: 00:16:3e:1f:9e:8c
  volatile.eth0.name: eth0
  volatile.idmap.base: "7000000"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":7000000,"Nsid":0,"Maprange":1000000},{"Isuid":false,"Isgid":true,"Hostid":7000000,"Nsid":0,"Maprange":1000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":7000000,"Nsid":0,"Maprange":1000000},{"Isuid":false,"Isgid":true,"Hostid":7000000,"Nsid":0,"Maprange":1000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":7000000,"Nsid":0,"Maprange":1000000},{"Isuid":false,"Isgid":true,"Hostid":7000000,"Nsid":0,"Maprange":1000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 51e5434f-3a02-4de2-adb7-a1bd223cca0d
  volatile.uuid.generation: 51e5434f-3a02-4de2-adb7-a1bd223cca0d
devices:
  eth0:
    ipv4.address: 10.128.0.25
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

[stgraber]

Can't reproduce it on the main branch and a recent OS (debian 12 with 6.12 kernel).
So this may be related to either a lack of pidfd or something to do with older filesystem APIs.

Going to try to reproduce this with the main branch on 20.04.

[stgraber]

Reproduced it. I had to use a system with a filesystem that doesn't support VFS idmap, in this case ZFS on 20.04 to cause it to happen.

[stgraber]

Yeah, almost certainly a pidfd issue as the same setup on 22.04, still ZFS, still no VFS idmap is working just fine.

[stgraber]

That's unfortunate because I can easily spit out new binaries for 22.04 but not so much for 20.04...
Going to see if I can't just downgrade the kernel of 22.04 to make debugging easier.

[stgraber]

I hate old kernels... We really need to bump our minimal kernel release up a bit...