[ubuntu][24.04][v3.4.0] ssh host key changed after update

[sdwilsh]

Kairos version:
CPU architecture, OS, and Version:
I'm filing this after getting everything back into a good state, so I don't have this, but I was using 24.04-standard-amd64-generic-v3.4.0 from the KAIROS_IMAGE_LABEL.

Describe the bug
After upgrading, get the dreaded host identification has changed error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

To Reproduce

1. Run quay.io/kairos/ubuntu:24.04-standard-amd64-generic-v3.3.1-k3sv1.32.1-k3s1
2. ssh into it to have the host key known
3. kairos-agent upgrade --source oci:quay.io/kairos/ubuntu:24.04-standard-amd64-generic-v3.4.0-k3s1.31.7-k3s1
4. reboot to complete upgrade
5. ssh in again, and see that the host key has changed

Expected behavior
ssh key should not change

[Itxaka]

Uh, this is an important one, thanks for the report. This is weird, I think the ssh folder is binded to persistent, so an upgrade should not affect the ssh keys in any way.

[Itxaka]

installed 3.3.1

got the following keys:

192.168.122.50 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMkuP1xJWcm3x1+ZfML4kXQSbSlxu2KZ6YWfk3R7UmLe
192.168.122.50 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCSE1xnWnvNuZ3FC/OhW8kC19pz5tffEk7c5ZOKSn5f2ir0cKd/UO0gLvSkhp+jdmWPLYnkOIgQH/52RYIZhvvYhJtf1BppAs34JL3BLknGlM4I49DwHWMzbN+J1iXsYAffhDHzApBGOJ5jKiEYBPFTafOYRFSjt6qOBAzc+ZE5mlfXYQXpRK5PFX8zRBD+K9J21mg5M4kIdpUMAw+UVMspo9oIn0d0zDCSf7WC3ZmCq7ChafTpqo+gHeZ84db2TyWs3xK4b7JqmDp4F93N6h4J8KkdA4RqEuBpbxCf51UKo5Ehv1TZJiAUwwvDLe6ogc6+5uw9awTHdDfVQ9V35QYDaHvKtADrlE4n9kzihqcPzNETmqyXSu3b6mjoVndefPTz+RXP9imPFbLdgpWp4OnDsHA7DYDnz3HtYYdMPPj+OC8hvA+G94lfCaxNbx5YgIg2KTvigDFxv5sdwNFbiGSzjZS9EVdCWAAYAoKGMxm3kL2ttH7wagMOiUsQznnECN8=
192.168.122.50 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOF+TjvEh+b/EsvK0IktFIBV3z9a1THnkvm3H9T+/un+1NNs5AErKTaKVp8AK7UkHa8cDJq8gOwWmogXrXueAiE=

which match the keys in the machine:

root@localhost:/etc/ssh# cat ssh_host_ed25519_key.pub 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMkuP1xJWcm3x1+ZfML4kXQSbSlxu2KZ6YWfk3R7UmLe root@localhost
root@localhost:/etc/ssh# cat ssh_host_ed25519_key.pub 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMkuP1xJWcm3x1+ZfML4kXQSbSlxu2KZ6YWfk3R7UmLe root@localhost
root@localhost:/etc/ssh# cat ssh_host_rsa_key.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCSE1xnWnvNuZ3FC/OhW8kC19pz5tffEk7c5ZOKSn5f2ir0cKd/UO0gLvSkhp+jdmWPLYnkOIgQH/52RYIZhvvYhJtf1BppAs34JL3BLknGlM4I49DwHWMzbN+J1iXsYAffhDHzApBGOJ5jKiEYBPFTafOYRFSjt6qOBAzc+ZE5mlfXYQXpRK5PFX8zRBD+K9J21mg5M4kIdpUMAw+UVMspo9oIn0d0zDCSf7WC3ZmCq7ChafTpqo+gHeZ84db2TyWs3xK4b7JqmDp4F93N6h4J8KkdA4RqEuBpbxCf51UKo5Ehv1TZJiAUwwvDLe6ogc6+5uw9awTHdDfVQ9V35QYDaHvKtADrlE4n9kzihqcPzNETmqyXSu3b6mjoVndefPTz+RXP9imPFbLdgpWp4OnDsHA7DYDnz3HtYYdMPPj+OC8hvA+G94lfCaxNbx5YgIg2KTvigDFxv5sdwNFbiGSzjZS9EVdCWAAYAoKGMxm3kL2ttH7wagMOiUsQznnECN8= root@localhost
root@localhost:/etc/ssh# cat ssh_host_ecdsa_key.pub 
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOF+TjvEh+b/EsvK0IktFIBV3z9a1THnkvm3H9T+/un+1NNs5AErKTaKVp8AK7UkHa8cDJq8gOwWmogXrXueAiE= root@localhost

/etc/ssh is binded to persistent:

root@localhost:/etc/ssh# df -h .
Filesystem                         Size  Used Avail Use% Mounted on
/dev/disk/by-label/COS_PERSISTENT   15G  976M   13G   8% /etc/ssh

/dev/vda5 on /etc/ssh type ext4 (rw,relatime)

root@localhost:/usr/local/.state/etc-ssh.bind# ls
moduli        ssh_host_ecdsa_key      ssh_host_ed25519_key.pub  sshd_config
ssh_config    ssh_host_ecdsa_key.pub  ssh_host_rsa_key          sshd_config.d
ssh_config.d  ssh_host_ed25519_key    ssh_host_rsa_key.pub

Will now upgrade and check

[Itxaka]

After upgrade, indeed, the ssh fingerprint changed:

$ ssh [email protected]
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:l5FZp7fpszmZRvVD5ZDi6YMaKZNRRcRFNaPVPB/B00Y.
Please contact your system administrator.
Add correct host key in /home/itxaka/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/itxaka/.ssh/known_hosts:9
Host key for 192.168.122.50 has changed and you have requested strict checking.
Host key verification failed.

[Itxaka]

somehow the key has changed to the buildkit sandbox, so the key that was generated during the artifact building with init has ended up in the persistent. Probably some cloud-config is copying it on boot

[Itxaka]

Yes, this is done by immucore, when it does a bind mount it first rsyncs the existing contents of the underlying mount as to not remove whats there, so in this case, it rsyncs the generated keys in the original image.

Its clear, kairos-init should clear the /etc/ssh dir on building.

[Itxaka]

kairos-io/kairos-init#67 to see the diff, will release init 0.4.6 to fix this

[Itxaka]

manually tested, installed 3.3.1, ssh into it, added the fingerprint then upgraded to latest master build with init 0.4.6. Rebooted, ssh into it, same fingerprint.

This is now on master so it should be fixed, will probably release a 3.4.1 patch version soon.