Storage Management Agent (SMA) is a service providing a gRPC interface for orchestrating SPDK applications. It's a standalone application that allows users to create and manage various types of devices (e.g. NVMe, virtio-blk, etc.). The major difference between SMA's API and the existing SPDK-RPC interface is that it's designed to abstract the low level details exposed by SPDK-RPCs, which enables it to be more easily consumed by orchestration frameworks, such as k8s or OpenStack. This is especially important for deployments on IPUs (Infrastructure Processing Unit), which usually require a lot of hardware-specific options.
The interface is defined in a protobuf files located in proto directory. The generic interface common to all types of devices is defined in sma.proto file, while device-specific options are defined in their separate files (e.g. nvme.proto for NVMe).
Currently, the interface consists of four methods. Additionally, it defines two main types of objects: volumes and devices. A volume is a representation of some storage media. It is equivalent to a SPDK bdev and/or an NVMe namespace and can exist even if it's not presented to the host system. A device is usually a virtual/physical PCIe function that is exposed to a host. It is capable of presenting one or more volumes (depending on the type of the device) to a host.
The following sections provide a high-level description of each method. For more details, consult the protobuf definitions.
This method creates a device. If a device with given parameters already exists, it becomes a no-op and returns a handle to that device.
Input:
volume: Volume parameters describing a volume to immediately attach to the created device. This field may be optional for some device types (e.g. NVMe), while it may be required for others (e.g. virtio-blk). Extending parameters with crypto attributes like encryption type, keys, tweak mode allows the user to configure crypto when attaching a volume to a device. Users must specify the crypto engine to use under crypto section in config. It is also possible to register out-of-tree crypto engines by inheriting from the CryptoEngine class.params: Device-specific parameters. The type of this structure determines the type of device to create.Output:
handle: Opaque handle identifying the device.This method deletes a device. Volumes that are still attached to a device being deleted will be automatically detached.
Input:
handle: Device handle obtained from CreateDevice.This method creates a volume and attaches it to a device exposing it to the host. It might lead to establishing a connection to remote storage target. However, this is not always the case, even if the volume is remote. For instance, if a volume describes an NVMe namespace, it might already be connected if another volume on the same subsystem was created previously. It may be unsupported by some types of devices (e.g. virtio-blk).
Input:
volume: Parameters describing the volume to attach. The type of this structure determines the method to create it (e.g. direct NVMe-oF connection, NVMe-oF through discovery service, iSCSI, etc.). Extending parameters with crypto attributes like type, keys, tweak mode allowing the user to configure crypto when attaching a volume to a device. Users must specify the crypto engine to use under crypto section in config. It is also possible to register out-of-tree crypto engines by inheriting from the CryptoEngine class.device_handle: Device handle obtained from CreateDevice.This method detaches a volume from a device making it unavailable to the host. It may be unsupported by some types of devices (e.g. virtio-blk).
Input:
volume_id: Volume UUID/GUID.device_handle: Device handle obtained from CreateDevice.This method configures QoS on a per-device or per-volume level. Not all QoS settings have to be supported by each device, so users can use GetQosCapabilities to get capabilities.
Input:
handle: Device handle obtained from CreateDevice.volume_id: Volume UUID/GUID. If this parameter is omitted, the QoS will be set up on the whole device (all volumes attached to that device will share QoS settings). Some device types might only support configuring QoS on per-device (volume_id must be empty) or per-volume level (volume_id cannot be empty). This information can be obtained by sending a GetQosCapabilities request.maximum: Maximum allowed IOPS/bandwidth values.This method queries supported QoS settings.
Input:
device_type: Type of a device to query for QoS capabilities.Output:
capabilities: QoS capabilities response including device/volume QoS limits like read IOPS, write IOPS, read/write IOPS, read bandwidth, write bandwidth, read/write bandwidth.In order to run SMA, SPDK needs to be configured with the --with-sma flag. Then, SMA can be started using a script located in scripts/sma.py. It requires a YAML configuration file that specifies which types of devices to service, as well as several other options (e.g. listen address, SPDK-RPC socket, etc.). Device types not listed in the configuration will be disabled and it won't be possible to manage them. Below is an example configuration enabling two device types (NVMe/vfiouser and vhost-blk):
SMA provides a way to load external plugins implementing support for specific device types. A plugin will be loaded if it's specified in the SMA_PLUGINS environment variable (multiple plugins are separated with a colon) or if it's specified in the plugins section of the config file. For example, the following two methods are equivalent:
Each plugin needs to be in the python search path (either in one of the default directories or added to PYTHONPATH).
A plugin is required to define a global variable called devices storing a list of classes deriving from spdk.sma.DeviceManager. This base class define the interface each device needs to implement. Additionally, each DeviceManager needs to define a unique name that will be used to identify it in config file as well as the name of the protocol it supports. There can be many DeviceManagers supporting the same protocol, but only one can be active at a time. The name of the protocol shall match the type specified in CreateDeviceRequest.params (e.g. "nvme", "virtio_blk", etc.), as it'll be used to select the DeviceManager to handle a gRPC request. Finally, a DeviceManager needs to implement the own_device() method returning a boolean value indicating whether a given device handle is owned by that DeviceManager.