Open
Description
When we run kola in the pipeline we run many many tests in a single run, but only 4 at a time (--parallel=4).
It appears, though, that some qemu processes are hanging around longer. For example:
core@coreos-ppc64le-builder:~$ sudo ss -lutnp | grep qemu
udp UNCONN 0 0 0.0.0.0:32771 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=143))
udp UNCONN 0 0 0.0.0.0:33353 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=163))
udp UNCONN 0 0 0.0.0.0:33513 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=185))
udp UNCONN 0 0 0.0.0.0:35914 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=122))
udp UNCONN 0 0 0.0.0.0:36387 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=152))
udp UNCONN 0 0 0.0.0.0:36766 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=133))
udp UNCONN 0 0 0.0.0.0:36842 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=169))
udp UNCONN 0 0 0.0.0.0:36847 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=108))
udp UNCONN 0 0 0.0.0.0:37014 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=118))
udp UNCONN 0 0 0.0.0.0:37312 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=114))
udp UNCONN 0 0 0.0.0.0:37328 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=156))
udp UNCONN 0 0 0.0.0.0:37600 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=165))
udp UNCONN 0 0 0.0.0.0:37746 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=145))
udp UNCONN 0 0 0.0.0.0:37902 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=107))
udp UNCONN 0 0 0.0.0.0:38864 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=168))
udp UNCONN 0 0 0.0.0.0:38962 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=110))
udp UNCONN 0 0 0.0.0.0:40629 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=112))
udp UNCONN 0 0 0.0.0.0:41679 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=159))
udp UNCONN 0 0 0.0.0.0:41731 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=142))
udp UNCONN 0 0 0.0.0.0:42216 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=170))
udp UNCONN 0 0 0.0.0.0:43546 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=149))
udp UNCONN 0 0 0.0.0.0:44382 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=121))
udp UNCONN 0 0 0.0.0.0:44477 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=146))
udp UNCONN 0 0 0.0.0.0:44969 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=106))
udp UNCONN 0 0 0.0.0.0:45113 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=135))
udp UNCONN 0 0 0.0.0.0:45711 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=30))
udp UNCONN 0 0 0.0.0.0:45852 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=130))
udp UNCONN 0 0 0.0.0.0:46064 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=126))
udp UNCONN 0 0 0.0.0.0:47320 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=154))
udp UNCONN 0 0 0.0.0.0:47657 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=131))
udp UNCONN 0 0 0.0.0.0:48291 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=187))
udp UNCONN 0 0 0.0.0.0:48428 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=148))
udp UNCONN 0 0 0.0.0.0:50286 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=155))
udp UNCONN 0 0 0.0.0.0:50475 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=140))
udp UNCONN 0 0 0.0.0.0:50478 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=120))
udp UNCONN 0 0 0.0.0.0:51084 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=164))
udp UNCONN 0 0 0.0.0.0:51150 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=158))
udp UNCONN 0 0 0.0.0.0:51467 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=160))
udp UNCONN 0 0 0.0.0.0:51639 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=115))
udp UNCONN 0 0 0.0.0.0:51643 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=162))
udp UNCONN 0 0 0.0.0.0:51867 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=138))
udp UNCONN 0 0 0.0.0.0:51911 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=134))
udp UNCONN 0 0 0.0.0.0:52978 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=144))
udp UNCONN 0 0 0.0.0.0:53949 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=171))
udp UNCONN 0 0 0.0.0.0:54155 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=129))
udp UNCONN 0 0 0.0.0.0:54285 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=166))
udp UNCONN 0 0 0.0.0.0:54296 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=167))
udp UNCONN 0 0 0.0.0.0:54434 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=137))
udp UNCONN 0 0 0.0.0.0:55749 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=189))
udp UNCONN 0 0 0.0.0.0:55831 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=113))
udp UNCONN 0 0 0.0.0.0:55904 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=139))
udp UNCONN 0 0 0.0.0.0:56132 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=111))
udp UNCONN 0 0 0.0.0.0:56266 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=182))
udp UNCONN 0 0 0.0.0.0:56434 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=151))
udp UNCONN 0 0 0.0.0.0:56636 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=141))
udp UNCONN 0 0 0.0.0.0:56930 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=153))
udp UNCONN 0 0 0.0.0.0:57442 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=117))
udp UNCONN 0 0 0.0.0.0:57848 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=136))
udp UNCONN 0 0 0.0.0.0:57890 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=150))
udp UNCONN 0 0 0.0.0.0:58370 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=116))
udp UNCONN 0 0 0.0.0.0:58738 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=157))
udp UNCONN 0 0 0.0.0.0:59310 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=161))
udp UNCONN 0 0 0.0.0.0:59466 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=128))
udp UNCONN 0 0 0.0.0.0:60703 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=119))
udp UNCONN 0 0 *:37513 *:* users:(("qemu-system-ppc",pid=1975486,fd=127))
udp UNCONN 0 0 *:39929 *:* users:(("qemu-system-ppc",pid=1975486,fd=125))
udp UNCONN 0 0 *:46748 *:* users:(("qemu-system-ppc",pid=1975486,fd=124))
udp UNCONN 0 0 *:49099 *:* users:(("qemu-system-ppc",pid=1975486,fd=132))
udp UNCONN 0 0 *:50749 *:* users:(("qemu-system-ppc",pid=1975486,fd=123))
udp UNCONN 0 0 *:54827 *:* users:(("qemu-system-ppc",pid=1975486,fd=147))
tcp LISTEN 0 1 127.0.0.1:33359 0.0.0.0:* users:(("qemu-system-ppc",pid=1975455,fd=19))
tcp LISTEN 0 1 127.0.0.1:33229 0.0.0.0:* users:(("qemu-system-ppc",pid=1975987,fd=19))
tcp LISTEN 0 1 127.0.0.1:38673 0.0.0.0:* users:(("qemu-system-ppc",pid=1975486,fd=19))
tcp LISTEN 0 1 127.0.0.1:41731 0.0.0.0:* users:(("qemu-system-ppc",pid=1975753,fd=19))
The majority of those are owned by the same process so there might be some opportunity for us to figure out how to clean those up, or maybe they are legitimate somehow?
The bigger problem however is that we had a port clash with the kdump.crash.nfs test. The test requires the use of the "Qemu Host" IP:Port 10.0.2.2:2049 so it can end up clashing if something else happens to be using that port already.
Something like this happened recently in the RHCOS pipeline:
[2025-05-30T14:53:17.900Z] + set -xeuo pipefail
[2025-05-30T14:53:17.900Z] ++ umask
[2025-05-30T14:53:17.900Z] + '[' 0022 = 0000 ']'
[2025-05-30T14:53:17.900Z] + cd /home/jenkins/agent/workspace/build-arch
[2025-05-30T14:53:17.900Z] + cosa kola run --rerun --allow-rerun-success=tags=needs-internet --build=latest --output-dir=/home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola --on-warn-failure-exit-77 --arch=ppc64le '--tag=!reprovision' '--tag=!openshift' --parallel=4
[2025-05-30T14:53:19.782Z] warning: /sys/fs/selinux appears to be mounted but should not be; enabling workaround
[2025-05-30T14:53:19.782Z] kola -p qemu --build latest run --rerun --allow-rerun-success=tags=needs-internet --on-warn-failure-exit-77 --arch=ppc64le --tag=!reprovision --tag=!openshift --parallel=4 --output-dir /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola
[2025-05-30T14:53:20.708Z] ⏭️ Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":
[2025-05-30T14:53:20.708Z] 👉 https://issues.redhat.com/browse/OCPBUGS-42688
[2025-05-30T14:53:20.708Z] ⏭️ Skipping kola test pattern "ext.config.shared.multipath.resilient":
[2025-05-30T14:53:20.708Z] 👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937
[2025-05-30T14:53:21.275Z] === RUN ostree.hotfix
[2025-05-30T14:53:21.275Z] === RUN coreos.unique.boot.failure
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.disks.lvmdevices
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.clhm.ignition-warnings
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.containers.quadlet
[2025-05-30T14:53:21.275Z] === RUN coreos.misc.disk.varlibcontainers
[2025-05-30T14:53:21.275Z] === RUN coreos.ignition.mount.disks
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.nameserver
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.boot.grub2-install
[2025-05-30T14:53:21.275Z] === RUN fips.enable
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ntp.chrony.dhcp-propagation
[2025-05-30T14:53:21.275Z] === RUN multipath.day1
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.systemd-enable-units
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.kargs
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.var-mount.simple
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.reboot
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ignition.delete-config
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.mtu-on-bond-ignition
[2025-05-30T14:53:21.275Z] === RUN coreos.ignition.sethostname
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file
[2025-05-30T14:53:21.275Z] === RUN fips.enable.partitions
[2025-05-30T14:53:21.275Z] === RUN rhcos.sssd
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.no-persist-ip
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.gshadow
[2025-05-30T14:53:21.275Z] === RUN rhcos.selinux.manage
[2025-05-30T14:53:21.275Z] === RUN coreos.unique.boot.ignition.failure
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.podman.rootless-systemd
[2025-05-30T14:53:21.275Z] === RUN non-exclusive-test-bucket-0
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.var-mount.luks
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.force-persist-ip
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.ssh.custom-host-key-permissions
[2025-05-30T14:53:21.275Z] === RUN ext.config.shared.networking.prefer-ignition-networking
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.groups
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.systemd.network-online
[2025-05-30T14:53:21.276Z] === RUN coreos.selinux.boolean
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.rd-net-timeout-carrier
[2025-05-30T14:53:21.276Z] === RUN ext.config.disk.nvme-compat-symlink
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.boot.bootupd-validate
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.systemd.enable-service
[2025-05-30T14:53:21.276Z] === RUN rpmostree.upgrade-rollback
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.stable-boot
[2025-05-30T14:53:21.276Z] === RUN kdump.crash.ssh
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.hostname.fallback-hostname
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.failure
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.v2.users
[2025-05-30T14:53:21.276Z] === RUN coreos.auth.verify
[2025-05-30T14:53:21.276Z] === RUN multipath.day2
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.nmstate.policy
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.remote
[2025-05-30T14:53:21.276Z] === RUN basic
[2025-05-30T14:53:21.276Z] === RUN rpmostree.status
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.team-dhcp-via-ignition
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.mtu-on-bond-kargs
[2025-05-30T14:53:21.276Z] === RUN ext.config.rpm-ostree.replace-rt-kernel
[2025-05-30T14:53:21.276Z] === RUN ostree.sync
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.selinux.enforcing
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation
[2025-05-30T14:53:21.276Z] === RUN ostree.remote
[2025-05-30T14:53:21.276Z] === RUN ext.config.systemd.journal-compat
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.once
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.symlink
[2025-05-30T14:53:21.276Z] === RUN ostree.unlock
[2025-05-30T14:53:21.276Z] === RUN multipath.partition
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.kargs-rd-net
[2025-05-30T14:53:21.276Z] === RUN rhcos.services-disabled
[2025-05-30T14:53:21.276Z] === RUN kdump.crash.nfs
[2025-05-30T14:53:21.276Z] === RUN ext.config.rpm-ostree.startlimit
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.ignition.resource.remote
[2025-05-30T14:53:21.276Z] === RUN rpmostree.install-uninstall
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.kdump.crash
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.no-default-initramfs-net-propagation.bootif
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.systemd.condition-needs-update
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.nmstate.state
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.var-mount.scsi-id
[2025-05-30T14:53:21.276Z] === RUN rhcos.network.multiple-nics
[2025-05-30T14:53:21.276Z] === RUN podman.base
[2025-05-30T14:53:21.276Z] === RUN rhcos.selinux.boolean.persist
[2025-05-30T14:53:21.276Z] === RUN ext.config.shared.networking.bridge-static-via-kargs
[2025-05-30T14:53:21.276Z] === RUN coreos.ignition.mount.partitions
[2025-05-30T14:53:59.909Z] === RUN ostree.hotfix/unlock
[2025-05-30T14:54:01.789Z] --- PASS: rpmostree.status (40.41s)
[2025-05-30T14:54:03.679Z] --- PASS: ext.config.rpm-ostree.startlimit (42.28s)
[2025-05-30T14:54:03.934Z] --- PASS: ext.config.shared.systemd.network-online (42.65s)
[2025-05-30T14:54:18.756Z] === RUN ostree.hotfix/install
[2025-05-30T14:54:18.756Z] === RUN ostree.hotfix/uninstall
[2025-05-30T14:54:19.010Z] === RUN ostree.hotfix/persist
[2025-05-30T14:54:40.871Z] === RUN basic/MachineID
[2025-05-30T14:54:41.125Z] === RUN basic/RHCOSGrowpart
[2025-05-30T14:54:41.380Z] === RUN basic/PortSSH
[2025-05-30T14:54:41.380Z] === RUN basic/DbusPerms
[2025-05-30T14:54:41.947Z] === RUN basic/ServicesActive
[2025-05-30T14:54:41.947Z] === RUN basic/ReadOnly
[2025-05-30T14:54:42.201Z] === RUN basic/Useradd
[2025-05-30T14:54:44.084Z] --- PASS: ext.config.shared.networking.hostname.fallback-hostname (42.11s)
[2025-05-30T14:54:45.450Z] --- PASS: basic (41.98s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/MachineID (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/RHCOSGrowpart (0.17s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/PortSSH (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/DbusPerms (0.47s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/ServicesActive (0.15s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/ReadOnly (0.14s)
[2025-05-30T14:54:45.450Z] --- PASS: basic/Useradd (0.26s)
[2025-05-30T14:54:53.518Z] === RUN ostree.hotfix/rollback
[2025-05-30T14:55:15.379Z] --- PASS: ext.config.shared.ignition.remote (68.46s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix (131.57s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/unlock (19.34s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/install (1.01s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/uninstall (0.58s)
[2025-05-30T14:55:33.410Z] --- PASS: ostree.hotfix/persist (33.57s)
[2025-05-30T14:55:33.411Z] --- PASS: ostree.hotfix/rollback (36.74s)
[2025-05-30T14:55:51.430Z] --- PASS: coreos.auth.verify (38.99s)
[2025-05-30T14:56:13.300Z] --- PASS: coreos.ignition.v2.users (40.33s)
[2025-05-30T14:56:16.560Z] --- PASS: coreos.ignition.failure (25.20s)
[2025-05-30T14:56:19.066Z] --- PASS: multipath.day2 (93.63s)
[2025-05-30T14:56:57.700Z] --- PASS: coreos.ignition.systemd.enable-service (39.70s)
[2025-05-30T14:57:07.623Z] --- PASS: ext.config.shared.networking.nmstate.policy (142.97s)
[2025-05-30T14:57:14.139Z] --- PASS: ext.config.shared.networking.mtu-on-bond-ignition (54.41s)
[2025-05-30T14:57:35.993Z] --- PASS: ext.config.shared.ignition.stable-boot (42.16s)
[2025-05-30T14:57:48.145Z] --- PASS: coreos.ignition.groups (39.44s)
[2025-05-30T14:57:50.649Z] === RUN rpmostree.upgrade-rollback/upgrade
[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.networking.prefer-ignition-networking (48.79s)
[2025-05-30T14:58:29.279Z] --- PASS: ext.config.shared.ssh.custom-host-key-permissions (42.20s)
[2025-05-30T14:58:51.152Z] === RUN rpmostree.upgrade-rollback/rollback
[2025-05-30T14:58:52.512Z] --- PASS: kdump.crash.ssh (155.81s)
[2025-05-30T14:58:52.512Z] cluster.go:151: Running as unit: run-r1a8a98404bfa4f99ba5db0a15aceba72.service
[2025-05-30T14:59:14.374Z] --- PASS: ext.config.shared.networking.force-persist-ip (49.33s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback (147.12s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback/upgrade (60.35s)
[2025-05-30T14:59:40.925Z] --- PASS: rpmostree.upgrade-rollback/rollback (46.89s)
[2025-05-30T15:00:12.931Z] --- PASS: ext.config.shared.var-mount.luks (101.53s)
[2025-05-30T15:00:39.405Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps
[2025-05-30T15:00:39.405Z] --- PASS: fips.enable (105.63s)
[2025-05-30T15:00:39.405Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.license
[2025-05-30T15:00:39.964Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.growfs
[2025-05-30T15:00:40.887Z] === RUN non-exclusive-test-bucket-0/ext.config.files.localtime
[2025-05-30T15:00:41.811Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink
[2025-05-30T15:00:42.369Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot
[2025-05-30T15:00:44.250Z] === RUN non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga
[2025-05-30T15:00:45.172Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks
[2025-05-30T15:00:59.991Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit
[2025-05-30T15:00:59.991Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables
[2025-05-30T15:01:03.255Z] === RUN non-exclusive-test-bucket-0/ext.config.files.conntrack-tools
[2025-05-30T15:01:03.815Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.sudoers
[2025-05-30T15:01:04.737Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme
[2025-05-30T15:01:06.618Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start
[2025-05-30T15:01:08.507Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service
[2025-05-30T15:01:09.867Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite
[2025-05-30T15:01:10.790Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.content-origins
[2025-05-30T15:01:12.671Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version
[2025-05-30T15:01:13.231Z] === RUN non-exclusive-test-bucket-0/ext.config.files.env-godebug
[2025-05-30T15:01:16.485Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links
[2025-05-30T15:01:17.045Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled
[2025-05-30T15:01:18.928Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking
[2025-05-30T15:01:25.454Z] --- PASS: ext.config.shared.reboot (74.49s)
[2025-05-30T15:01:27.974Z] === RUN non-exclusive-test-bucket-0/ext.config.files.composefs-maybe
[2025-05-30T15:01:28.228Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.podman.dns
[2025-05-30T15:01:31.495Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota
[2025-05-30T15:01:32.056Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service
[2025-05-30T15:01:32.614Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log
[2025-05-30T15:01:34.497Z] --- PASS: ext.config.shared.ignition.delete-config (141.08s)
[2025-05-30T15:01:34.497Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default
[2025-05-30T15:01:35.419Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.setuid
[2025-05-30T15:01:37.301Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client
[2025-05-30T15:01:37.861Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir
[2025-05-30T15:01:38.785Z] === RUN non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service
[2025-05-30T15:01:39.345Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro
[2025-05-30T15:01:40.270Z] === RUN non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service
[2025-05-30T15:01:41.201Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service
[2025-05-30T15:01:41.761Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.logging.printk
[2025-05-30T15:01:42.684Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change
[2025-05-30T15:01:44.567Z] === RUN non-exclusive-test-bucket-0/ext.config.ssh.config
[2025-05-30T15:01:46.447Z] === RUN non-exclusive-test-bucket-0/ext.config.version.rhel-major-version
[2025-05-30T15:01:56.372Z] --- PASS: coreos.unique.boot.ignition.failure (21.50s)
[2025-05-30T15:02:00.528Z] --- PASS: ext.config.shared.podman.rootless-systemd (82.57s)
[2025-05-30T15:02:02.410Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config
[2025-05-30T15:02:02.969Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info
[2025-05-30T15:02:03.893Z] === RUN non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp
[2025-05-30T15:02:04.451Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context
[2025-05-30T15:02:05.374Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts
[2025-05-30T15:02:07.257Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.console-config
[2025-05-30T15:02:08.624Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.tls
[2025-05-30T15:02:15.158Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs
[2025-05-30T15:02:15.412Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read
[2025-05-30T15:02:17.292Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions
[2025-05-30T15:02:19.173Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions
[2025-05-30T15:02:20.531Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy
[2025-05-30T15:02:25.763Z] === RUN non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder
[2025-05-30T15:02:26.322Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.setgid
[2025-05-30T15:02:28.203Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context
[2025-05-30T15:02:36.276Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match
[2025-05-30T15:02:38.157Z] --- PASS: ext.config.shared.var-mount.simple (73.45s)
[2025-05-30T15:02:50.312Z] --- PASS: ext.config.shared.ignition.kargs (54.48s)
[2025-05-30T15:03:05.133Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin
[2025-05-30T15:03:06.493Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.root-bash
[2025-05-30T15:03:07.415Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables
[2025-05-30T15:03:08.337Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming
[2025-05-30T15:03:10.217Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust
[2025-05-30T15:03:12.096Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd
[2025-05-30T15:03:13.979Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers
[2025-05-30T15:03:14.537Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression
[2025-05-30T15:03:15.463Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch
[2025-05-30T15:03:17.972Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.files.system-generators
[2025-05-30T15:03:19.330Z] --- PASS: ext.config.shared.ignition.systemd-enable-units (41.43s)
[2025-05-30T15:03:19.887Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.kdump.service
[2025-05-30T15:03:20.811Z] --- PASS: rhcos.selinux.manage (80.02s)
[2025-05-30T15:03:20.811Z] === RUN non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0 (224.07s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.rpm-ostree.container-deps (1.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.license (0.74s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.growfs (0.82s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.localtime (0.75s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.check-symlink (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.no-systemd-firstboot (1.85s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.check-qemu-ga (0.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.validate-symlinks (12.47s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-immutable-bit (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.executables (4.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.conntrack-tools (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.sudoers (0.75s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.partition-scheme (1.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-start (1.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.logrotate-service (1.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.rpmdb-sqlite (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.content-origins (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.aleph-version (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.env-godebug (2.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.no-google-device-links (0.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.composefs.enabled (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.rootless-pasta-networking (8.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.composefs-maybe (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.podman.dns (2.82s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-prjquota (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.network-online-service (0.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ignition.journald-log (1.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.no-default-initramfs-net-propagation.default (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setuid (1.81s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-dhcp-client (0.78s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.yum-repo-dir (0.77s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.afterburn.sshkey-service (0.79s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.root-boot-ro (0.76s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.console.console-login-helper-service (0.80s)
[2025-05-30T15:03:24.963Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.systemd-repart-service (0.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.logging.printk (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.default-network-behavior-change (1.85s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.ssh.config (1.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.version.rhel-major-version (15.93s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.ntp.chrony.coreos-platform-chrony-config (0.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.clhm.network-device-info (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.version.drop-azure-ptp (0.75s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.usrlocal-context (0.74s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.systemd.default-unit-timeouts (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.console-config (1.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.tls (5.80s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.disks.tmpfs (0.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.systemd-env-read (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.etc-permissions (1.76s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.file-directory-permissions (1.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.unmodified-policy (4.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.files.kata-osbuilder (0.77s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.setgid (1.78s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.podman-tmpfs-context (7.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.selinux.file-context-policy-match (28.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nm-ifcfg-rh-plugin (1.86s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.root-bash (0.79s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.firewall.iptables (0.84s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.networking.nic-naming (1.85s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.security.coreos-update-ca-trust (1.82s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.boot.bootupd (1.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.kernel-headers (0.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.initrd.compression (0.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kubernetes.kube-watch (2.86s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.files.system-generators (1.81s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.kdump.service (0.83s)
[2025-05-30T15:03:24.964Z] --- PASS: non-exclusive-test-bucket-0/ext.config.shared.swap.zram-default (0.90s)
[2025-05-30T15:03:33.032Z] --- PASS: ext.config.shared.gshadow (42.37s)
[2025-05-30T15:04:11.662Z] --- PASS: ext.config.shared.networking.no-persist-ip (48.25s)
[2025-05-30T15:04:13.543Z] --- PASS: rhcos.sssd (40.81s)
[2025-05-30T15:04:52.177Z] --- PASS: ext.config.shared.ntp.chrony.dhcp-propagation (86.90s)
[2025-05-30T15:04:54.683Z] --- PASS: multipath.day1 (95.28s)
[2025-05-30T15:05:41.270Z] --- PASS: coreos.ignition.sethostname (39.65s)
[2025-05-30T15:05:42.628Z] --- PASS: ext.config.shared.var-mount.scsi-id (51.02s)
[2025-05-30T15:05:49.154Z] --- PASS: ext.config.shared.networking.ifname-karg.everyboot-systemd-link-file (95.74s)
[2025-05-30T15:05:59.077Z] --- PASS: fips.enable.partitions (108.57s)
[2025-05-30T15:06:31.074Z] --- PASS: ext.config.shared.networking.bridge-static-via-kargs (48.44s)
[2025-05-30T15:06:33.583Z] === RUN podman.base/info
[2025-05-30T15:06:34.505Z] === RUN podman.base/resources
[2025-05-30T15:06:49.331Z] --- PASS: podman.base (50.21s)
[2025-05-30T15:06:49.331Z] --- PASS: podman.base/info (1.07s)
[2025-05-30T15:06:49.331Z] --- PASS: podman.base/resources (9.93s)
[2025-05-30T15:06:49.331Z] --- PASS: coreos.ignition.mount.partitions (73.95s)
[2025-05-30T15:06:50.254Z] --- FAIL: kdump.crash.nfs (2.02s)
[2025-05-30T15:06:50.254Z] kdump.go:291: listen tcp :2049: bind: address already in use
[2025-05-30T15:07:05.075Z] --- PASS: rhcos.selinux.boolean.persist (75.01s)
[2025-05-30T15:07:26.941Z] --- PASS: ext.config.systemd.journal-compat (52.94s)
[2025-05-30T15:07:41.760Z] === RUN ostree.unlock/unlock
[2025-05-30T15:07:42.016Z] === RUN ostree.unlock/install
[2025-05-30T15:07:43.376Z] === RUN ostree.unlock/uninstall
[2025-05-30T15:07:43.934Z] === RUN ostree.unlock/discard
[2025-05-30T15:08:01.956Z] === RUN rhcos.services-disabled/ServicesDisabled
[2025-05-30T15:08:03.840Z] --- PASS: multipath.partition (73.31s)
[2025-05-30T15:08:04.398Z] --- PASS: rhcos.services-disabled (40.27s)
[2025-05-30T15:08:04.398Z] --- PASS: rhcos.services-disabled/ServicesDisabled (0.40s)
[2025-05-30T15:08:22.421Z] --- PASS: ostree.unlock (76.70s)
[2025-05-30T15:08:22.421Z] --- PASS: ostree.unlock/unlock (1.03s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/install (1.34s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/uninstall (0.67s)
[2025-05-30T15:08:22.422Z] --- PASS: ostree.unlock/discard (33.76s)
[2025-05-30T15:08:24.302Z] --- PASS: rhcos.network.multiple-nics (96.31s)
[2025-05-30T15:08:46.163Z] --- PASS: coreos.ignition.symlink (39.76s)
[2025-05-30T15:08:52.687Z] --- PASS: ext.config.shared.networking.kargs-rd-net (48.23s)
[2025-05-30T15:09:14.561Z] --- PASS: ext.config.shared.networking.no-default-initramfs-net-propagation.bootif (50.21s)
[2025-05-30T15:09:32.584Z] --- PASS: coreos.ignition.once (71.58s)
[2025-05-30T15:09:59.059Z] --- PASS: ext.config.shared.boot.grub2-install (43.53s)
[2025-05-30T15:09:59.059Z] --- PASS: coreos.misc.disk.varlibcontainers (75.83s)
[2025-05-30T15:09:59.059Z] cluster.go:151: Created symlink /etc/systemd/system/var-lib-containers.mount.wants/[email protected] → /etc/systemd/system/[email protected].
[2025-05-30T15:09:59.059Z] cluster.go:151: Created symlink /etc/systemd/system/local-fs.target.wants/var-lib-containers.mount → /etc/systemd/system/var-lib-containers.mount.
[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nameserver (48.43s)
[2025-05-30T15:10:55.197Z] --- PASS: ext.config.shared.networking.nmstate.state (75.73s)
[2025-05-30T15:11:03.279Z] --- PASS: ext.config.shared.systemd.condition-needs-update (130.35s)
[2025-05-30T15:11:13.206Z] --- PASS: coreos.ignition.mount.disks (72.43s)
[2025-05-30T15:11:19.736Z] === RUN rpmostree.install-uninstall/install
[2025-05-30T15:11:29.662Z] --- PASS: ext.config.shared.clhm.ignition-warnings (40.83s)
[2025-05-30T15:11:56.145Z] --- PASS: ext.config.shared.containers.quadlet (44.55s)
[2025-05-30T15:12:08.297Z] --- PASS: ext.config.shared.ignition.resource.remote (38.49s)
[2025-05-30T15:12:09.657Z] --- FAIL: ostree.sync (2.02s)
[2025-05-30T15:12:09.657Z] sync.go:117: listen tcp :2049: bind: address already in use
[2025-05-30T15:12:36.136Z] === RUN rpmostree.install-uninstall/uninstall
[2025-05-30T15:13:14.783Z] --- PASS: ext.config.shared.disks.lvmdevices (75.45s)
[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.networking.ifname-karg.udev-rule-firstboot-propagation (80.97s)
[2025-05-30T15:13:32.806Z] --- PASS: ext.config.shared.kdump.crash (149.09s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall (176.30s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall/install (74.08s)
[2025-05-30T15:13:42.730Z] --- PASS: rpmostree.install-uninstall/uninstall (65.40s)
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/add
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/list
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/show-url
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/refs
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/summary
[2025-05-30T15:14:09.212Z] === RUN ostree.remote/delete
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote (41.97s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/add (0.33s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/list (0.13s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/show-url (0.25s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/refs (0.23s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/summary (0.52s)
[2025-05-30T15:14:12.471Z] --- PASS: ostree.remote/delete (0.51s)
[2025-05-30T15:14:22.406Z] --- PASS: ext.config.shared.networking.mtu-on-bond-kargs (49.03s)
[2025-05-30T15:14:25.663Z] --- PASS: ext.config.shared.selinux.enforcing (73.70s)
[2025-05-30T15:14:25.663Z] mke2fs 1.47.2 (1-Jan-2025)
[2025-05-30T15:14:30.896Z] --- PASS: ext.config.shared.networking.team-dhcp-via-ignition (48.24s)
[2025-05-30T15:14:45.724Z] --- PASS: coreos.unique.boot.failure (18.75s)
[2025-05-30T15:15:03.744Z] --- PASS: ext.config.disk.nvme-compat-symlink (41.07s)
[2025-05-30T15:15:30.215Z] --- PASS: ext.config.shared.boot.bootupd-validate (59.30s)
[2025-05-30T15:16:16.824Z] --- PASS: ext.config.shared.networking.rd-net-timeout-carrier (86.07s)
[2025-05-30T15:16:16.824Z] --- PASS: coreos.selinux.boolean (71.74s)
[2025-05-30T15:18:53.236Z] KVM: Failed to create TCE64 table for liobn 0x80000001
[2025-05-30T15:18:59.774Z] --- PASS: ext.config.rpm-ostree.replace-rt-kernel (287.08s)
[2025-05-30T15:18:59.774Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z] ======== Re-running failed tests (flake detection) ========
[2025-05-30T15:18:59.774Z]
[2025-05-30T15:18:59.774Z] ⏭️ Skipping kola test pattern "ext.config.version.rhaos-pkgs-match-openshift":
[2025-05-30T15:18:59.774Z] 👉 https://issues.redhat.com/browse/OCPBUGS-42688
[2025-05-30T15:18:59.774Z] ⏭️ Skipping kola test pattern "ext.config.shared.multipath.resilient":
[2025-05-30T15:18:59.774Z] 👉 https://github.com/coreos/fedora-coreos-tracker/issues/1937
[2025-05-30T15:19:00.028Z] === RUN ostree.sync
[2025-05-30T15:19:00.028Z] === RUN kdump.crash.nfs
[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'
[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)
[2025-05-30T15:19:32.587Z] sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused
[2025-05-30T15:21:38.968Z] --- PASS: kdump.crash.nfs (157.10s)
[2025-05-30T15:21:38.968Z] cluster.go:151: Running as unit: run-rbc26bb21f6424c438e84e7781f60c9e3.service
[2025-05-30T15:21:38.968Z] FAIL, output in /home/jenkins/agent/workspace/build-arch/tmp/kola-Wndtl/kola/rerun
[2025-05-30T15:21:38.968Z] Error: harness: test suite failed
[2025-05-30T15:21:38.968Z] 2025-05-30T15:21:37Z cli: harness: test suite failed
[2025-05-30T15:21:38.968Z] failed to execute cmd-kola: exit status 1
[2025-05-30T15:19:00.585Z] qemu-system-ppc64: -netdev user,id=eth0,hostfwd=tcp:127.0.0.1:36287-:22,hostfwd=tcp:127.0.0.1:2049-:2049,hostname=qemu0: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'
[2025-05-30T15:19:32.587Z] --- FAIL: ostree.sync (31.54s)
[2025-05-30T15:19:32.587Z] sync.go:117: failed to establish qmp connection: dial unix /var/tmp/mantle-qemu2099425295/qmp-1748618340436644144.sock: connect: connection refused
Basically for whatever reason the host clashed: Could not set up host forwarding rule 'tcp:127.0.0.1:2049-:2049'. I don't know if another kdump.crash.nfs test was running for another pipeline (or if they would even clash).
You can also reproduce something similar with cosa kola run kdump.crash.nfs --multiply 2 --parallel 2.
Metadata
Metadata
Assignees
Labels
No labels