VFIO-USER: A new virtualization protocol

We’re excited to announce support for NVMe over vfio-user, a technology that allows SPDK to present fully emulated NVMe devices into virtual machines. Virtual machines can leverage their existing NVMe drivers to communicate with the devices and data is transferred efficiently to or from SPDK using shared memory. In other words, just like vhost-user but able to emulate NVMe devices instead of virtio-blk or virtio-scsi devices.

Standardization and support for vfio-user is currently underway in the QEMU community. The draft specification has all interested parties aligned and is maturing quickly. The protocol itself is capable of emulating any physical device, not just NVMe, but emulating NVMe devices with SPDK has been the first and primary consumer of the new interface so far.

Nutanix has released a convenience library that helps implement a server for the protocol and SPDK leverages this in its implementation.

The NVMe device emulation is implemented using SPDK’s existing NVMe-oF target, treating vfio-user as a shared memory “transport” in the same way it supports TCP, RDMA, PCIe, and Fibre Channel. The NVMe-oF specification already happened to require extensive emulation of an NVMe device to implement, so almost all of the required logic to emulate a physical NVMe SSD was already present. It really was as simple as adding a new transport plugin for vfio-user. While this transport is not officially part of the NVMe-oF specification, the transport plugin system has been set up to allow for custom transports such as this. These transport plugins can even exist in shared libraries outside of the SPDK repository and discovered at run time, although the vfio-user transport is built in to SPDK itself.

We’re excited about this advance in virtualization technology. By emulating a physical NVMe device, any operating system with an NVMe driver (i.e. all of them) can talk to the device. This is especially important for Windows - no more loading virtio drivers! It also allows for virtual machines to boot from the emulated NVMe devices because the UEFI BIOS contains an NVMe driver.

Long term, we fully expect vfio-user to gain market share not only in the storage industry, but also in networking and elsewhere. We expect the performance to be excellent - at least as good as virtio-blk but likely better - and we’ll be excited to share benchmarks as soon as they’re available.