Parallel and Distributed Fuzzing

Relevant source files

The following files were used as context for generating this wiki page:

• src/afl-fuzz.c
• utils/distributed_fuzzing/sync_script.sh
• src/afl-fuzz-run.c
• src/afl-fuzz-stats.c
• README.md
• nyx_mode/README.md
• config.h
• frida_mode/MapDensity.md

AFL++ is designed to leverage multiple CPU cores and even multiple machines to accelerate the fuzzing process. This is achieved through a parallel fuzzing mechanism that allows multiple fuzzer instances to collaborate. The core idea is that different instances can explore different parts of the program's state space simultaneously and share interesting findings (testcases that discover new coverage) with each other. This collaborative approach significantly increases the overall efficiency and speed of bug discovery.

The parallelization is based on a Main/Secondary architecture. One instance is designated as the "Main" node, which has slightly different behaviors and often acts as a central point for certain strategies, while all other instances are "Secondary" nodes. These instances periodically synchronize their queues, ensuring that valuable testcases are propagated throughout the entire fuzzing campaign. AFL++ also supports syncing with foreign fuzzers and has specialized modes for distributed fuzzing with virtual machine-based targets like Nyx.

Sources: src/afl-fuzz.c:321-324, README.md:100-102

Core Architecture

AFL++'s parallel fuzzing relies on a Main-Secondary model, where fuzzer instances are configured with different roles using the -M and -S command-line options.

Sources: src/afl-fuzz.c:1012-1083

Main Node (-M)

A single fuzzer instance in a campaign should be designated as the Main node. This is done using the -M <sync_id> option.

• Role: The Main node is responsible for more than just fuzzing. It typically handles tasks like syncing with foreign fuzzers.
• Behavioral Changes:
  • Trimming Disabled: The Main node does not trim testcases (afl->disable_trim = 1).
  • Sequential Queue Selection: It uses a sequential, non-randomized method for selecting the next seed from the queue (afl->old_seed_selection = 1). This ensures a more deterministic progression through the corpus.
  • Recommended Power Schedule: It is recommended to use fast or explore power schedules for Main nodes.

If no -M or -S option is provided, afl-fuzz will automatically configure itself as a Secondary node with a sync_id of "default".

Sources: src/afl-fuzz.c:1012-1043, src/afl-fuzz.c:1662-1667

Secondary Node (-S)

All other fuzzer instances in the campaign are designated as Secondary nodes using the -S <sync_id> option.

• Role: Secondary nodes are the workhorses of the fuzzing campaign. Their primary focus is to fuzz and find new paths.
• Synchronization: Secondary nodes typically sync only from the Main node to receive new testcases, while the Main node syncs from all other nodes. This creates a hub-and-spoke model for testcase distribution.

Sources: src/afl-fuzz.c:1045-1065, src/afl-fuzz-run.c:690-704

Foreign Fuzzer Syncing (-F)

AFL++ can synchronize its findings with other, non-AFL++ fuzzers. This is configured on the Main node.

• Usage: The -F <path> option specifies a directory where a foreign fuzzer's queue is located. This option can be used multiple times.
• Requirement: This option can only be used with a Main (-M) instance.

Sources: src/afl-fuzz.c:1067-1083

Configuration

Parallel fuzzing is configured through a combination of command-line options and environment variables.

Command-Line Options

The following table summarizes the key command-line options for setting up parallel and distributed fuzzing.

Option	Argument	Description
-M	id[:N/M]	Designates the instance as a Main node with a given id. Optionally, can specify a node ID and total nodes for distributed work.
-S	id	Designates the instance as a Secondary node with a given id.
-F	path	Syncs with a foreign fuzzer queue directory. Requires -M and can be specified up to FOREIGN_SYNCS_MAX times.
-Y	(none)	Enables Nyx distributed mode, which has a special Main/Secondary configuration.

Sources: src/afl-fuzz.c:321-324, src/afl-fuzz.c:1012-1083, src/afl-fuzz.c:1346-1355

Environment Variables

Several environment variables can be used to fine-tune the synchronization behavior.

Variable	Description
AFL_SYNC_TIME	Overrides the default sync time between fuzzer instances (in minutes).
AFL_IMPORT_FIRST	If set, the fuzzer will perform an initial sync to import testcases from other instances before starting its own fuzzing cycle.
AFL_FINAL_SYNC	If set, performs one last sync when the fuzzer is about to exit. This can delay the exit.
AFL_NO_SYNC	Disables all synchronization with other fuzzer instances.

Sources: src/afl-fuzz.c:467-468

Configuration Constants

The default synchronization behavior is defined by constants in config.h.

Constant	Default Value	Description
SYNC_INTERVAL	8	The number of havoc cycles between sync attempts.
SYNC_TIME	20 * 60 * 1000	The minimum time in milliseconds between syncs. This is halved for Main nodes.
FOREIGN_SYNCS_MAX	4	The maximum number of foreign fuzzers that can be synced with using the -F flag.

Sources: config.h:352-359, src/afl-fuzz.c:324

Synchronization Process

The synchronization process allows fuzzer instances to share interesting testcases. This is managed by the sync_fuzzers function.

The overall flow is as follows:

1. The fuzzer instance opens its sync_dir (the output directory).
2. It iterates through the subdirectories, each representing another fuzzer instance.
3. It skips its own directory and, if it's a secondary node, any other secondary nodes (syncing only from the main).
4. For each valid peer, it reads the last synced testcase ID from its local .synced/<peer_id> file.
5. It then scans the peer's queue directory for new testcases with IDs greater than the last synced ID.
6. Each new testcase is read, executed, and if it produces interesting coverage, saved to the current fuzzer's queue.
7. After processing, the .synced/<peer_id> file is updated with the new latest ID.

To prevent re-importing testcases that originated from the current instance, the is_known_case function checks the filename format. Synced files are typically named id:xxxxxx,sync:<origin_id>,..., and this function checks if <origin_id> matches the current fuzzer's sync_id.

Sources: src/afl-fuzz-run.c:667-880, src/afl-fuzz-run.c:600-631

This diagram illustrates the synchronization logic for a fuzzer instance.

Specialized Parallel Modes

Nyx Mode (-Y)

Nyx mode, a VM-based fuzzing engine, uses a specialized parallel fuzzing setup activated by the -Y flag.

• Main Instance: The first instance must be started with -M 0. This instance creates the initial VM snapshot that will be shared with all secondary instances.
• Secondary Instances: Subsequent instances are started with -S <id>, where <id> is a unique, increasing number starting from 1 (e.g., -S 1, -S 2).
• Snapshot Reuse: To speed up the startup of analysis tools like afl-tmin or afl-analyze in a Nyx campaign, the AFL_NYX_REUSE_SNAPSHOT environment variable can be set to the path of an existing snapshot directory (e.g., out_dir/workdir/snapshot/).

This setup is mandatory for running Nyx in parallel. Using the standard -M/-S with string-based IDs is not supported.

Sources: nyx_mode/README.md:83-100, nyx_mode/README.md:108-120, src/afl-fuzz.c:1703-1733

Frida Mode Map Density

When fuzzing in parallel, Frida mode can use a different seed for its hashing function in each instance. This causes each instance to assign different IDs to code blocks and edges. The benefit is that if two edges collide in the coverage map for one fuzzer instance, it is unlikely they will collide in another. This diversity, combined with the collaborative nature of parallel fuzzing, helps overcome roadblocks caused by map collisions.

Sources: frida_mode/MapDensity.md:105-121

Distributed Fuzzing Across Machines

To run a fuzzing campaign across multiple machines, the synchronization directories must be shared. The utils/distributed_fuzzing/sync_script.sh script provides a practical example of how to achieve this using rsync and ssh.

The script performs the following actions:

1. Defines a list of remote hosts.
2. Defines the path to the fuzzer output directory on both the local and remote machines.
3. Uses rsync to pull queue directories from remote fuzzers to the local machine.
4. Uses rsync to push the local main fuzzer's queue to all remote machines.

This allows a main node on one machine to collect findings from secondary nodes on other machines and distribute its own findings back to them.

# Example snippet from sync_script.sh
# ...
# rsync remote -> local
for host in "${hosts[@]}"; do
  if [ "$host" == "$HOSTNAME" ]; then
    continue
  fi
  echo "[*] Syncing from $host"
  for dir in $fuzzer_dirs; do
    if [ -d "$fuzzer_path/$dir" ]; then
      rsync -avz --delete "$host:$fuzzer_path/$dir/queue/" "$fuzzer_path/$dir/queue/"
    fi
  done
done

# rsync local -> remote
for host in "${hosts[@]}"; do
  if [ "$host" == "$HOSTNAME" ]; then
    continue
  fi
  echo "[*] Syncing to $host"
  rsync -avz --delete "$fuzzer_path/$main_fuzzer_dir/queue/" "$host:$fuzzer_path/$main_fuzzer_dir/queue/"
done

Sources: utils/distributed_fuzzing/sync_script.sh

Conclusion

Parallel and distributed fuzzing is a cornerstone feature of AFL++, enabling massive scalability to improve bug-finding capabilities. By using a Main/Secondary architecture, instances can efficiently share interesting testcases, ensuring that progress made by one fuzzer benefits the entire campaign. The system is highly configurable through command-line flags and environment variables, and it supports both multi-core setups on a single machine and distributed campaigns across a network. Specialized modes like Nyx and Frida further enhance parallel fuzzing with unique optimizations for jejich respective environments.