Nyx Mode for Snapshot Fuzzing
Relevant source files
The following files were used as context for generating this wiki page:Nyx is a full-system emulation fuzzing mode integrated into AFL++ that provides high-performance snapshot fuzzing capabilities. It is built on KVM and a modified version of QEMU, allowing it to fuzz both source-instrumented and binary-only targets within a complete guest OS environment. This mode is particularly recommended for targets that cannot be fuzzed efficiently in the default fork mode or persistent mode.
Nyx operates exclusively on Linux and is currently restricted to the x86_64 architecture, with aarch64 support in development. For source-based fuzzing, it requires a modern Linux kernel (5.11+). For binary-only fuzzing, it leverages Intel Processor Trace (PT) and requires an Intel CPU (6th gen or newer) along with a special 5.10 kernel.
Sources: nyx_mode/README.md:3-16
Architecture and Components
Nyx mode's architecture is composed of several key components that work together to provide full-system snapshot fuzzing. The main afl-fuzz process communicates with a specialized version of QEMU via libnyx to manage the virtual machine, execute test cases, and retrieve coverage information.
Sources: nyx_mode/build_nyx_support.sh:31-41, src/afl-fuzz-init.c:1348-1356
The diagram below illustrates the high-level architecture of Nyx mode.
The core components are managed as Git submodules within the AFL++ repository:
- QEMU-Nyx: A fork of QEMU with modifications for snapshotting and fuzzing.
- packer: A toolset for packaging the target binary and its dependencies into a format consumable by Nyx.
- libnyx: A Rust library that acts as the intermediary between
afl-fuzzand QEMU-Nyx.
Sources: nyx_mode/build_nyx_support.sh:31-41, nyx_mode/update_ref.sh:8-78
Build and Setup
Building Nyx mode requires specific dependencies and a multi-step process orchestrated by a build script.
Dependencies
Before building, ensure the following packages are installed in addition to the standard AFL++ dependencies:
libgtk-3-devpax-utilspython3-msgpackpython3-jinja2libcapstone-dev- A recent Rust compiler toolchain (installed via
rustup).
Sources: nyx_mode/README.md:21-29
Build Process
The build is handled by the build_nyx_support.sh script. This script performs sanity checks, initializes the required Git submodules (QEMU-Nyx, packer, libnyx), and compiles each component.
The following diagram shows the flow of the build script:
Sources: nyx_mode/build_nyx_support.sh
To build Nyx mode, run the following from the nyx_mode directory:
./build_nyx_support.sh
This script will also be executed as part of the top-level make distrib command.
Sources: nyx_mode/README.md:35-38
Target Preparation
Targets for Nyx mode must be specially packaged into a "share directory" which contains the binary, its dependencies, and configuration files.
Packaging with nyx_packer.py
The nyx_packer.py script automates the creation of the share directory. It analyzes the target binary, collects its dependencies, and sets up a bootstrap script.
python3 nyx_mode/packer/packer/nyx_packer.py \
/PATH/TO/TARGET \
PACKAGE-DIRECTORY \
afl \
instrumentation \
--fast_reload_mode \
--purge
Sources: nyx_mode/README.md:52-61
The key parameters for the packer are summarized below:
| Parameter | Description | Source |
|---|---|---|
/PATH/TO/TARGET | Path to the target executable. | nyx_mode/README.md:53 |
PACKAGE-DIRECTORY | The output directory for the Nyx package. | nyx_mode/README.md:54 |
afl / spec | The agent type. afl is for AFL++ fuzzing. | nyx_mode/README.md:154-156 |
instrumentation / processor_trace | The coverage mode. instrumentation for compile-time, processor_trace for Intel PT. | nyx_mode/README.md:56, nyx_mode/README.md:159-161 |
-file /path/to/file | Specifies that fuzzer input is passed via a file. | nyx_mode/README.md:63-64 |
-args "..." | Command-line arguments for the target. | nyx_mode/README.md:66-67 |
--fast_reload_mode | Enables fast snapshots to improve performance. | nyx_mode/README.md:166-167 |
--purge | Purges the output directory before packaging. | nyx_mode/README.md:59 |
Configuration with nyx_config_gen.py
After packaging, a Nyx configuration file (config.ron) must be generated. This file contains VM settings and other parameters for the Nyx runtime.
python3 nyx_mode/packer/packer/nyx_config_gen.py PACKAGE-DIRECTORY Kernel
Sources: nyx_mode/README.md:72-74, test/test-nyx-mode.sh:23-24
Running the Fuzzer
With the Nyx package ready, you can start fuzzing.
KVM Module Setup
Nyx relies on KVM. The kernel modules must be loaded with a specific parameter to enable the backdoor hypercalls used for communication.
sudo modprobe -r kvm-intel
sudo modprobe -r kvm
sudo modprobe kvm enable_vmware_backdoor=y
sudo modprobe kvm-intel
Sources: nyx_mode/README.md:79-84
Standalone Mode (-X)
For a single fuzzer instance, use the -X flag. The target for afl-fuzz is the package directory created by the packer.
afl-fuzz -i in -o out -X -- ./PACKAGE-DIRECTORY
Sources: src/afl-fuzz.c:1068-1075, nyx_mode/README.md:89-91
Parallel Mode (-Y)
For parallel fuzzing, Nyx has a special multi-processor mode enabled with the -Y flag. This is different from the standard AFL++ -M/-S mode for fork-based fuzzing.
- A main instance must be started with
-M 0. This instance creates the initial snapshot. - Secondary instances are started with
-S <id>, where<id>is a number greater than 0.
# Main instance
afl-fuzz -i in -o out -Y -M 0 -- ./PACKAGE-DIRECTORY
# Secondary instance
afl-fuzz -i in -o out -Y -S 1 -- ./PACKAGE-DIRECTORY
Sources: src/afl-fuzz.c:1077-1085, nyx_mode/READMEmd:100-104
The following diagram shows the startup sequence for parallel mode.
Configuration
Nyx mode is configured through command-line flags and environment variables.
Command-Line Flags
| Flag | afl-fuzz context | Description | Source |
|---|---|---|---|
-X | Standalone | Enables Nyx mode for a single fuzzer instance. | src/afl-fuzz.c:1068-1075 |
-Y | Parallel | Enables Nyx mode for distributed fuzzing with a main/secondary setup. | src/afl-fuzz.c:1077-1085 |
Environment Variables
| Variable | Description | Default | Source |
|---|---|---|---|
AFL_NYX_AUX_SIZE | Size of the shared memory region for crash reports and hprintf. Must be a multiple of 4096. | 4096 bytes | nyx_mode/README.md:250-255 |
AFL_NYX_DISABLE_SNAPSHOT_MODE | Disables fast snapshots, except for crashes. | Not set | nyx_mode/README.md:221-222 |
AFL_NYX_REUSE_SNAPSHOT | Path to an existing snapshot directory to speed up the launch of companion tools. | Not set | nyx_mode/README.md:130-132 |
AFL_NYX_LOG | Redirects Nyx hprintf messages to a specified file. | Not set | src/afl-fuzz.c:426 |
TMPDIR | Base directory for creating the Nyx temporary workdir. | /tmp | src/afl-common.c:989-991 |
Advanced Features
Crash Reporting
When a crash occurs, the Nyx agent can pass detailed information to AFL++ through a shared memory region called the "Nyx Auxiliary Buffer". This information is saved in a .log file alongside the crashing input. For the default agent, this includes the faulting address and signal number. If the target is compiled with ASan, the full ASan report is included. The size of this buffer can be increased with AFL_NYX_AUX_SIZE.
Sources: nyx_mode/README.md:239-255
Custom Harnesses
For complex targets, a custom harness or agent may be necessary. The file nyx_mode/custom_harness/example.c provides a template for this. A custom harness communicates with the host (QEMU-Nyx) via hypercalls, which are exposed as simple command-line tools within the guest.
| Tool | Description |
|---|---|
hcat | Copies a string from the guest to the host. |
hget | Requests a file from the host's share directory. |
hget_bulk | An optimized version of hget for large files. |
habort | Sends an abort signal to the host, useful for bootstrap failures. |
hpush | Transfers a file from the guest to the host's dump/ directory. |
Sources: nyx_mode/README.md:180-193, nyx_mode/custom_harness/example.c
The example harness demonstrates how to implement a basic coverage map and communicate with the fuzzer without using the pre-built agent.
// From nyx_mode/custom_harness/example.c
// ...
/* set coverage bitmap */
page_aligned_bitmap[0] = 1;
/* tell the hypervisor that we are done */
hypercall(NYX_SET_DONE, 0);
// ...
page_aligned_bitmap[1] = 1;
// ...
Sources: nyx_mode/custom_harness/example.c:104-110
Using Companion Tools
AFL++ companion tools like afl-tmin, afl-cmin, and afl-analyze support Nyx mode. To use them, simply add the -X flag and provide the path to the Nyx package directory.
afl-tmin -i in_file -o out_file -X -- ./PACKAGE-DIRECTORY
The startup performance of these tools can be significantly improved by reusing a snapshot created by a parallel fuzzing job (-Y -M 0). Set the AFL_NYX_REUSE_SNAPSHOT environment variable to the path of the snapshot directory (out_dir/workdir/snapshot/).
Sources: nyx_mode/README.md:118-135
Conclusion
Nyx mode is a powerful extension to AFL++ that enables high-speed, snapshot-based fuzzing of entire systems. By leveraging KVM and a custom QEMU build, it provides a robust environment for testing complex targets that are difficult to fuzz with traditional methods. Its support for both source-instrumented and binary-only targets, combined with a flexible packaging and configuration system, makes it a valuable tool for security researchers.