An Overview of SPDK Applications

SPDK is primarily a development kit that delivers libraries and header files for use in other applications. However, SPDK also contains a number of applications. These applications are primarily used to test the libraries, but many are full featured and high quality. The major applications in SPDK are:

There are also a number of tools and examples in the examples directory.

The SPDK targets are all based on a common framework so they have much in common. The framework defines a concept called a subsystem and all functionality is implemented in various subsystems. Subsystems have a unified initialization and teardown path.

Configuring SPDK Applications

Command Line Parameters

The SPDK application framework defines a set of base command line flags for all applications that use it. Specific applications may implement additional flags.

Param Long Param Type Default Description
-c –config string Configuration file
-d –limit-coredump flag false Limit coredump
-e –tpoint-group-mask integer 0x0 Tracepoint group mask
-g –single-file-segments flag Create just one hugetlbfs file
-h –help flag show all available parameters and exit
-i –shm-id integer Multi process mode
-m –cpumask CPU mask 0x1 application CPU mask
-n –mem-channels integer all channels number of memory channels used for DPDK
-p –master-core integer first core in CPU mask master (primary) core for DPDK
-r –rpc-socket string /var/tmp/spdk.sock RPC listen address
-s –mem-size integer all hugepage memory Memory size
–silence-noticelog flag disable notice level logging to stderr
-u –no-pci flag Disable PCI access.
–wait-for-rpc flag Deferred initialization
-B –pci-blacklist B:D:F PCI address blacklist and whitelist.
-W –pci-whitelist B:D:F PCI address blacklist and whitelist.
-R –huge-unlink flag Unlink hugepage files after initialization
–huge-dir string the first discovered allocate hugepages from a specific mount
-L –logflag string Debug log

Configuration file

Historically, the SPDK applications were configured using a configuration file. This is still supported, but is considered deprecated in favor of JSON RPC configuration. See JSON-RPC Methods for details.

Note that --config and --wait-for-rpc cannot be used at the same time.

Limit coredump

By default, an SPDK application will set resource limits for core file sizes to RLIM_INFINITY. Specifying --limit-coredump will not set the resource limits.

Tracepoint group mask

SPDK has an experimental low overhead tracing framework. Tracepoints in this framework are organized into tracepoint groups. By default, all tracepoint groups are disabled. --tpoint-group-mask can be used to enable a specific subset of tracepoint groups in the application.

Note: Additional documentation on the tracepoint framework is in progress.

Deferred initialization

SPDK applications progress through a set of states beginning with STARTUP and ending with RUNTIME.

If the --wait-for-rpc parameter is provided SPDK will pause just before starting subsystem initialization. This state is called STARTUP. The JSON RPC server is ready but only a small subsystem of commands are available to set up initialization parameters. Those parameters can't be changed after the SPDK application enters RUNTIME state. When the client finishes configuring the SPDK subsystems it needs to issue the start_subsystem_init RPC command to begin the initialization process. After rpc_start_subsystem_init returns true SPDK will enter the RUNTIME state and the list of available commands becomes much larger.

To see which RPC methods are available in the current state, issue the get_rpc_methods with the parameter current set to true.

For more details see JSON-RPC Methods documentation.

Create just one hugetlbfs file

Instead of creating one hugetlbfs file per page, this option makes SPDK create one file per hugepages per socket. This is needed for Virtio driver to be used with more than 8 hugepages. See 2MB hugepages.

Multi process mode

When --shm-id is specified, the application is started in multi-process mode. Applications using the same shm-id share their memory and NVMe devices. The first app to start with a given id becomes a primary process, with the rest, called secondary processes, only attaching to it. When the primary process exits, the secondary ones continue to operate, but no new processes can be attached at this point. All processes within the same shm-id group must use the same --single-file-segments setting.

Memory size

Total size of the hugepage memory to reserve. If DPDK env layer is used, it will reserve memory from all available hugetlbfs mounts, starting with the one with the highest page size. This option accepts a number of bytes with a possible binary prefix, e.g. 1024, 1024M, 1G. The default unit is megabyte.

Starting with DPDK 18.05.1, it's possible to reserve hugepages at runtime, meaning that SPDK application can be started with 0 pre-reserved memory. Unlike hugepages pre-reserved at the application startup, the hugepages reserved at runtime will be released to the system as soon as they're no longer used.

Disable PCI access

If SPDK is run with PCI access disabled it won't detect any PCI devices. This includes primarily NVMe and IOAT devices. Also, the VFIO and UIO kernel modules are not required in this mode.

PCI address blacklist and whitelist

If blacklist is used, then all devices with the provided PCI address will be ignored. If a whitelist is used, only whitelisted devices will be probed. -B or -W can be used more than once, but cannot be mixed together. That is, -B and -W cannot be used at the same time.

Unlink hugepage files after initialization

By default, each DPDK-based application tries to remove any orphaned hugetlbfs files during its initialization. This option removes hugetlbfs files of the current process as soon as they're created, but is not compatible with --shm-id.

Debug log

Enable a specific debug log type. This option can be used more than once. A list of all available types is provided in the --help output, with --logflag all enabling all of them. Debug logs are only available in debug builds of SPDK.

CPU mask

Whenever the CPU mask is mentioned it is a string in one of the following formats:

  • Case insensitive hexadecimal string with or without "0x" prefix.
  • Comma separated list of CPUs or list of CPU ranges. Use '-' to define range.

Example

The following CPU masks are equal and correspond to CPUs 0, 1, 2, 8, 9, 10, 11 and 12:

0x1f07
0x1F07
1f07
[0,1,2,8-12]
[0, 1, 2, 8, 9, 10, 11, 12]