The Storage Performance Development Kit iSCSI target application is named iscsi_tgt. This following section describes how to run iscsi from your cloned package.
This guide starts by assuming that you can already build the standard SPDK distribution on your platform.
Once built, the binary will be in app/iscsi_tgt.
If you want to kill the application by using signal, make sure use the SIGTERM, then the application will release all the shared memory resource before exit, the SIGKILL will make the shared memory resource have no chance to be released by applications, you may need to release the resource manually.
The following diagram shows relations between different parts of iSCSI structure described in this document.
Configuring iSCSI Target via config file
A iscsi_tgt specific configuration file is used to configure the iSCSI target. A fully documented example configuration file is located at etc/spdk/iscsi.conf.in.
The configuration file is used to configure the SPDK iSCSI target. This file defines the following: TCP ports to use as iSCSI portals; general iSCSI parameters; initiator names and addresses to allow access to iSCSI target nodes; number and types of storage backends to export over iSCSI LUNs; iSCSI target node mappings between portal groups, initiator groups, and LUNs.
You should make a copy of the example configuration file, modify it to suit your environment, and then run the iscsi_tgt application and pass it the configuration file using the -c option. Right now, the target requires elevated privileges (root) to run.
app/iscsi_tgt/iscsi_tgt -c /path/to/iscsi.conf
Assigning CPU Cores to the iSCSI Target
SPDK uses the DPDK Environment Abstraction Layer to gain access to hardware resources such as huge memory pages and CPU core(s). DPDK EAL provides functions to assign threads to specific cores. To ensure the SPDK iSCSI target has the best performance, place the NICs and the NVMe devices on the same NUMA node and configure the target to run on CPU cores associated with that node. The following command line option is used to configure the SPDK iSCSI target:
This is a hexadecimal bit mask of the CPU cores where the iSCSI target will start polling threads. In this example, CPU cores 24, 25, 26 and 27 would be used.
Configuring a LUN in the iSCSI Target
Each LUN in an iSCSI target node is associated with an SPDK block device. See Block Device User Guide for details on configuring SPDK block devices. The block device to LUN mappings are specified in the configuration file as:
This exports a malloc'd target. The disk is a RAM disk that is a chunk of memory allocated by iscsi in user space. It will use offload engine to do the copy job instead of memcpy if the system has enough DMA channels.
Configuring iSCSI Target via RPC method
In addition to the configuration file, the iSCSI target may also be configured via JSON-RPC calls. See JSON-RPC Methods for details.
add_portal_group – Add a portal group.
delete_portal_group – Delete an existing portal group.
add_pg_ig_maps – Add initiator group to portal group mappings to an existing iSCSI target node.
delete_pg_ig_maps – Delete initiator group to portal group mappings from an existing iSCSI target node.
get_portal_groups – Show information about all available portal groups.
At the iSCSI level, we provide the following support for Hotplug:
bdev/nvme: At the bdev/nvme level, we start one hotplug monitor which will call spdk_nvme_probe() periodically to get the hotplug events. We provide the private attach_cb and remove_cb for spdk_nvme_probe(). For the attach_cb, we will create the block device base on the NVMe device attached, and for the remove_cb, we will unregister the block device, which will also notify the upper level stack (for iSCSI target, the upper level stack is scsi/lun) to handle the hot-remove event.
scsi/lun: When the LUN receive the hot-remove notification from block device layer, the LUN will be marked as removed, and all the IOs after this point will return with check condition status. Then the LUN starts one poller which will wait for all the commands which have already been submitted to block device to return back; after all the commands return back, the LUN will be deleted.
Known bugs and limitations
For write command, if you want to test hotplug with write command which will cause r2t, for example 1M size IO, it will crash the iscsi tgt. For read command, if you want to test hotplug with large read IO, for example 1M size IO, it will probably crash the iscsi tgt.