Fuzzing is a software testing technique that incorporates feeding random, unexpected input to a computer program to test unexpected paths of software execution. The SPDK project uses two approaches to fuzz testing:
Brute force randomization of inputs
LLVM coverage-guided randomization
The SPDK fuzzer application tries to fuzz the NVMe-oF target or a physical NVMe drive by submitting
randomized NVMe commands through the SPDK NVMe initiator. To achieve this, NVMe commands are
sent through the
spdk_nvme_ctrlr_cmd_io_raw interface, or in the case of admin commands
spdk_nvme_ctrlr_cmd_admin_raw. The data in those commands is completely random,
generated using the current time as a seed.
This application saves generated commands in a JSON formatted file that allows for further debugging. But using random input has some drawbacks: the fuzzer can spend lots of time testing inputs that do not increase coverage and even after a long run, there may still be code paths left untested.
For more information take a look at
To overcome this issue, another tool to fuzz test SPDK apps is used. LibFuzzer is a coverage guided fuzzer that comes from the LLVM project. Every input sent to SPDK is also tested against code coverage data to see if it increased. If a random input increases code coverage, it is saved and used as the basis for further mutation, allowing to check nodes placed deeper in the execution tree and to save time on testing inputs that can be rejected early.
nvme_fuzz this application submits NVMe commands through the SPDK NVMe initiator,
but this time random input is provided by
LLVMFuzzerRunDriver and used to perform the chosen test
sudo CC=clang-13 CXX=clang++-13 ./configure --with-fuzzer='/usr/lib/llvm-13/lib/clang/13.0.1/lib/linux/libclang_rt.fuzzer_no_main-x86_64.a'
sudo ./test/fuzz/llvm/nvmf/run.sh 1
Since LibFuzzer is a part of the LLVM project, to compile this application, it’s necessary
CXX variables to clang and provide the path to the fuzzer library.
The LLVM fuzzer is also used to test the
vfio-user transport. This application sends
unexpected input to mimic misbehaving and/or malicious virtual machines using the
sudo CC=clang-13 CXX=clang++-13 ./configure --with-fuzzer='/usr/lib/llvm-13/lib/clang/13.0.1/lib/linux/libclang_rt.fuzzer_no_main-x86_64.a' --with-vfio-user
sudo ./test/fuzz/llvm/vfio/run.sh 1
Running fuzz tests
An important advantage of the LLVM fuzzer is the corpus file. Each random input generated by the fuzzer that explores a new path is saved, allowing resumption of testing from the point where it was stopped and avoiding lost progress.
First existing data from a corpus is processed to check if it is still valid and then fuzzing continues from the last random input. It is also possible to provide predetermined data to recreate issues found by the fuzzer or to start from valid samples.
The SPDK CI system has two jobs,
long-fuzz-vfio that utilize persistent
This way every run is a continuation of the previous one, continually increasing code coverage, which can be observed by looking at coverage data expose in the job artifacts.
To try it for yourself just set
SPDK_TEST_FUZZER and all tests should start in parallel.
sudo SPDK_TEST_FUZZER=1 ./test/fuzz/llvm/vfio/run.sh 1
For more information about LLVM Fuzzer please visit LibFuzzer
Thanks for reading!