error: deleting cgroup

[zimbatm]

There might be some issue with the nix cgroup implementation.

Source: https://buildbot.nix-community.org/#/builders/1390/builds/161

nixos-disk-image>          Specify the 'raw' format explicitly to remove the restrictions.
nixos-disk-image> EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
error: deleting cgroup '"/sys/fs/cgroup/system.slice/nix-daemon.service/nix-build-uid-872546304"': Device or resource busy
program finished with exit code 1

This needs a bit of debugging.

SSH-ing into build03, it looks like the cgroup was deleted successfully:

[root@build03:~]# ls -la /sys/fs/cgroup/system.slice/nix-daemon.service/ | grep nix
drwxr-xr-x  2 root root 0 Dec 23 14:30 nix-daemon

[zimbatm]

https://buildbot.nix-community.org/#/builders/1384/builds/162

nixos-disk-image>          Specify the 'raw' format explicitly to remove the restrictions.
nixos-disk-image> EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
warning: killing stray builder process 1316157 ()...
warning: killing stray builder process 1337725 ()...
warning: killing stray builder process 1337726 ()...
error: deleting cgroup '"/sys/fs/cgroup/system.slice/nix-daemon.service/nix-build-uid-873005056"': Device or resource busy

https://github.com/nixos/nix/blob/f1187cb696584739884687d788a6fbb4dd36c61c/src/libutil/linux/cgroup.cc#L96

[zowoq]

Looks like it is the same as #1459 (comment), hitting the buildbot 20 minute timeout and getting cgroup errors.

[Mic92]

Potential fix: Mic92/nix-1@fa1f677

[Mic92]

Unsure if this is causing nix builds to hang. Still debugging.

[Mic92]

Ok. Can't reproduce, but I suspect those were old builds prior to the patch or some vm test related hangs.

[Mic92]

NixOS/nix#13135

[fzakaria]

We tried a reproduction (but failed) but here it is for posterity:

seq 1 100 | xargs -P 16 -I"{}"  nix build -f cgroup-reproduction.nix --extra-experimental-features "cgroups" --use-cgroups --max-jobs 50 -L --argstr "random" "{}"

{random}: let
  pkgs = import <nixpkgs> {};
in
  with pkgs;
    derivation {
      name = "cgroup-reproduction-${random}-${(toString builtins.currentTime)}";
      system = builtins.currentSystem;
      builder = "${bash}/bin/bash";
      args = [
        (writeText "script" ''
          (${pkgs.util-linux}/bin/setsid ${pkgs.coreutils}/bin/sleep infinity &)
          wait $!
          ${pkgs.coreutils}/bin/cat /proc/self/cgroup
          # ${pkgs.coreutils}/bin/sleep 10;
        '')
      ];
    }