[Devices] virtio vsock

[raduweiss]

We want to have vsock support to enable container integration, but we don't want to use vhost since that would be another attack surface to directly exposes the host kernel. Instead, we'll write another back-end for vsock. See #194.

Currenty, virtio vsock exists in the codebase as an experimental development-only rust feature, with the vhost implementation.

[dhrgit]

Updating this issue with the current state of affairs, as discussed internally.

The main focus is on maintaining the Firecracker security barrier. I.e. Firecracker must control all data exchanged between the guest and the host. Using the stock vsock-via-vhost mechanism would bypass this barrier, allowing a malicious guest to pass data directly to the host kernel.

Currently I see two possible approaches to have both vsock and our security barrier:

1. Emulate a vsock device via a different back-end on the host (e.g. a unix socket). That would mean the guest end would use AF_VSOCK, while the host end AF_UNIX.
   • pro: vhost is not used at all, so no new attack surface is added
   • con: added complexity to making AF_UNIX and AF_VSOCK work together
   • con: non-standard vsock use; requires the user to write more Firecracker-specific code
2. Use vhost as back-end for an emulated vsock device, with a twist. I.e. implement both device-specific and driver-specific behavior in Firecracker, such that both the guest and vhost think they're taking to each other, while they're actually talking to Firecracker. This might work, since vhost doesn't depend on KVM, but needs further investigating / testing.
   • pro: standard AF_VSOCK functionality on both ends; no Firecracker-specific code required from the user
   • con: still using vhost code (new attack surface); all data fed into vhost is strictly controlled by Firecracker, though

I'm inclined towards the latter and I'll need to test it, since it's just a hypothesis, at the moment. Will update here in a couple of weeks.

[rn]

An example of option 1 can be found in https://github.com/moby/hyperkit/blob/master/src/lib/pci_virtio_sock.c

[dhrgit]

Update: internal discussion has settled on approach no 1. The guest end would be an emulated AF_VSOCK socket, while the host end would be provided via an AF_LOCAL / AF_UNIX socket.

This will require establishing some convention to support vsock ports, such as appending the port number to the host-end unix socket file path (e.g. /path/to/host_socket.sock:52).

Also, from the host end, this will require at least one listener (one socket) per guest, since each Firecracker VMM is designed to be isolated and independent.

If anybody tracking this issue has any input, please feel free to chime in. There are details not decided upon yet, so we'd appreciate any input into making this feature easier to integrate for Firecracker users.

[sboeuf]

@dhrgit
Trying to make sure I understand the proposal here. With the chosen approach no.1, you will basically implement some sort of translation/abstraction layer in Firecracker, from AF_UNIX to AF_VSOCK.
By AF_VSOCK emulation in the guest, I guess you mean you are thinking about writing a virtio driver that would communicate with the Firecracker backend, and that would expose an AF_VSOCK socket from the guest perspective, is that right?

I don't have a good grasp on potential performance issues related to this approach, but it'd be nice to include @stefanha in this discussion since he's vsock maintainer, and I'm sure he could give some valuable input here.

[stefanha]

Approach #1 is comparable to port forwarding. It's fine if you control all applications and the protocols they speak. If you wish to support existing applications then you may be forced to make invasive changes to those applications:

When communication involves exchanging network address information over an existing connection, the host application receiving AF_VSOCK address information would be confused because that address isn't directly usable with AF_LOCAL.

Also, managing port forwarding complicates things if you wish to allow users to add their own applications but is easy with a fixed set of applications that you control.

[dhrgit]

@sboeuf
There shouldn't be a need for any vsock driver changes. I meant Firecracker would emulate a vsock device, that works with the guest vsock driver. So, nothing changes on the guest/driver end.

On the host side, though, the only way to support AF_VSOCK would be via vhost. Since the dominant opinion is to avoid adding vhost as a new dependency (and attack surface), we need to find another solution. The proposed approach (to use an AF_UNIX socket) is what hyperkit does and is linked to above by @rn.

This does impose some limitations, since there's no perfect translation between AF_VSOCK and AF_UNIX. Both the host apps and the guest apps will need to be aware of these limitations, in order to establish a communication channel.

The implementation details are not yet set, so my proposal is to use this issue here to discuss them and come up with a solution that makes sense to everyone.

[samuelkarp]

Is it one AF_UNIX socket per guest, or one AF_UNIX socket per AF_VSOCK port? I think either would work for firecracker-containerd, but we expect to use multiple ports and would need to handle multiplexing if there's only a single AF_UNIX socket per guest.

[sboeuf]

@dhrgit @samuelkarp

@mcastelino and I talked about the approach you're about to take. From a Kata Containers perspective, it should be fine as we can tweak the application, kata-runtime and kata-shim in this case to handle the AF_UNIX socket instead of the regular AF_VSOCK one.

Now, from an implementation perspective, we were discussing how there were two distincts use cases that you need to handle depending if the application running inside the VM runs as:

• Server: the virtio_vsock device will be the one running the server exposing an AF_UNIX socket to the host. The core logic will retrieve the vsock frames through the virtqueues, and will run one AF_UNIX socket server per vsock port used in the VM. I can imagine the user of Firecracker specifying the guest CID and a root directory path where to create the sockets. You will need some sort of convention so that the user knows how to find the created socket name inside this directory where you could have several sockets, all related to a specific port. Another approach might be to let the user create a virtio_vsock device per CID:port pair, in which case the socket path could be also fully provided by the user.
• Client: This case means we expect to run the server on the host side. The user will have to provide a full socket path where it expects to listen from, and this will expose to the guest a single port.

Globally, you will proxy everything through the virtio_vsock device.

Let me know if I'm missing something or if I actually misunderstood parts of the design.