Fuzzing is a software testing technique that incorporates feeding random, unexpected input to a computer program to test unexpected paths of software execution. The SPDK project uses two approaches to fuzz testing:

• Brute force randomization of inputs

• LLVM coverage-guided randomization

NVMe fuzz

The SPDK fuzzer application tries to fuzz the NVMe-oF target or a physical NVMe drive by submitting randomized NVMe commands through the SPDK NVMe initiator. To achieve this, NVMe commands are sent through the spdk_nvme_ctrlr_cmd_io_raw interface, or in the case of admin commands spdk_nvme_ctrlr_cmd_admin_raw. The data in those commands is completely random, generated using the current time as a seed.

This application saves generated commands in a JSON formatted file that allows for further debugging. But using random input has some drawbacks: the fuzzer can spend lots of time testing inputs that do not increase coverage and even after a long run, there may still be code paths left untested.

For more information take a look at test/app/fuzz/nvme_fuzz/README.md

LLVM libfuzzer

To overcome this issue, another tool to fuzz test SPDK apps is used. LibFuzzer is a coverage guided fuzzer that comes from the LLVM project. Every input sent to SPDK is also tested against code coverage data to see if it increased. If a random input increases code coverage, it is saved and used as the basis for further mutation, allowing to check nodes placed deeper in the execution tree and to save time on testing inputs that can be rejected early.

llvm_nvme_fuzz

Similar to nvme_fuzz this application submits NVMe commands through the SPDK NVMe initiator, but this time random input is provided by LLVMFuzzerRunDriver and used to perform the chosen test defined in g_fuzzers array.

sudo CC=clang-13 CXX=clang++-13 ./configure --with-fuzzer='/usr/lib/llvm-13/lib/clang/13.0.1/lib/linux/libclang_rt.fuzzer_no_main-x86_64.a'
sudo ./test/fuzz/llvm/nvmf/run.sh 1

Since LibFuzzer is a part of the LLVM project, to compile this application, it’s necessary to set CC and CXX variables to clang and provide the path to the fuzzer library.

llvm_vfio_fuzz

The LLVM fuzzer is also used to test the vfio-user transport. This application sends unexpected input to mimic misbehaving and/or malicious virtual machines using the spdk_vfio_user_pci_bar_access interface.

sudo CC=clang-13 CXX=clang++-13 ./configure --with-fuzzer='/usr/lib/llvm-13/lib/clang/13.0.1/lib/linux/libclang_rt.fuzzer_no_main-x86_64.a' --with-vfio-user
sudo ./test/fuzz/llvm/vfio/run.sh 1

Running fuzz tests

An important advantage of the LLVM fuzzer is the corpus file. Each random input generated by the fuzzer that explores a new path is saved, allowing resumption of testing from the point where it was stopped and avoiding lost progress.

First existing data from a corpus is processed to check if it is still valid and then fuzzing continues from the last random input. It is also possible to provide predetermined data to recreate issues found by the fuzzer or to start from valid samples.

The SPDK CI system has two jobs, long-fuzz-nvmf and long-fuzz-vfio that utilize persistent corpus files.

This way every run is a continuation of the previous one, continually increasing code coverage, which can be observed by looking at coverage data expose in the job artifacts.

To try it for yourself just set SPDK_TEST_FUZZER and all tests should start in parallel.

sudo SPDK_TEST_FUZZER=1 ./test/fuzz/llvm/vfio/run.sh 1

For more information about LLVM Fuzzer please visit LibFuzzer

Thanks for reading!