Custom Mutators API

Relevant source files

The following files were used as context for generating this wiki page:

• custom_mutators/README.md
• src/afl-fuzz-mutators.c
• include/afl-fuzz.h
• custom_mutators/rust/custom_mutator/src/lib.rs
• test/test-custom-mutators.sh
• custom_mutators/examples/README.md
• custom_mutators/honggfuzz/honggfuzz.c
• custom_mutators/gramatron/gramfuzz.c
• custom_mutators/zmq/README.md
• custom_mutators/rust/README.md

The Custom Mutator API in AFL++ provides a powerful mechanism to extend and customize the fuzzing strategies employed by afl-fuzz. By implementing a specific interface, developers can create their own mutation logic, which can be tailored to complex data formats, protocols, or specific targets. These custom mutators can be written in C/C++, Python, or Rust and are loaded at runtime as shared libraries or modules.

This allows for the integration of structure-aware fuzzing, grammar-based generation, or even porting mutation strategies from other fuzzers like Honggfuzz and libFuzzer. Custom mutators can either augment the default AFL++ havoc and deterministic stages or replace them entirely, offering a high degree of flexibility.

Sources: custom_mutators/README.md, include/afl-fuzz.h:1229-1522

Enabling Custom Mutators

Custom mutators are enabled via environment variables. AFL++ supports loading mutators from C/C++ shared libraries and Python modules.

Sources: custom_mutators/README.md, test/test-custom-mutators.sh

C/C++ Shared Libraries

To use a custom mutator compiled as a shared library (.so file), set the AFL_CUSTOM_MUTATOR_LIBRARY environment variable.

AFL_CUSTOM_MUTATOR_LIBRARY=/path/to/your/mutator.so afl-fuzz ...

Multiple libraries can be loaded by separating their paths with a colon (:) or semicolon (;).

AFL_CUSTOM_MUTATOR_LIBRARY="./libmutator1.so:./libmutator2.so" afl-fuzz ...

Sources: custom_mutators/README.md, src/afl-fuzz-mutators.c:75-103, test/test-custom-mutators.sh:39-41

Python Modules

For Python-based mutators, set the AFL_PYTHON_MODULE environment variable to the name of the Python module. The module must be available in the PYTHONPATH.

export PYTHONPATH=/path/to/your/python_mutator_dir
export AFL_PYTHON_MODULE=your_mutator_module
afl-fuzz ...

Sources: include/afl-fuzz.h:558, test/test-custom-mutators.sh:70-73

Exclusive Mode

To use only the custom mutator(s) and disable AFL++'s built-in mutation strategies, set AFL_CUSTOM_MUTATOR_ONLY=1.

AFL_CUSTOM_MUTATOR_ONLY=1 AFL_CUSTOM_MUTATOR_LIBRARY=./mutator.so afl-fuzz ...

Sources: custom_mutators/README.md, include/afl-fuzz.h:549

Architecture and Lifecycle

The custom mutator is managed within afl-fuzz through the custom_mutator struct. This struct holds pointers to the functions implemented by the custom library, a handle to the loaded library (dlopen handle), and any private data the mutator needs.

Sources: include/afl-fuzz.h:1229-1522, src/afl-fuzz-mutators.c

Loading and Initialization

The following diagram illustrates how a custom mutator is loaded and initialized by afl-fuzz.

This diagram shows the initialization flow for both C/C++ and Python custom mutators.

Sources: src/afl-fuzz-mutators.c:60-141

Fuzzing Lifecycle

Once initialized, the mutator's functions are called at different points in the fuzzing loop. The core fuzzing interaction involves the afl_custom_fuzz function.

This flowchart shows the sequence of calls for a single input from the queue during the custom mutation stage.

Sources: include/afl-fuzz.h:1229-1522, src/afl-fuzz-mutators.c

Data Structures

The core data structures are defined in include/afl-fuzz.h.

struct custom_mutator: This structure holds all function pointers for the custom mutator API.

// Simplified from include/afl-fuzz.h
struct custom_mutator {
  const char *name;
  void       *dh; // dlopen handle
  void       *data; // custom mutator data ptr

  // API function pointers
  void *(*afl_custom_init)(afl_state_t *afl, unsigned int seed);
  size_t (*afl_custom_fuzz)(void *data, u8 *buf, size_t buf_size, u8 **out_buf,
                            u8 *add_buf, size_t add_buf_size, size_t max_size);
  u32 (*afl_custom_fuzz_count)(void *data, const u8 *buf, size_t buf_size);
  size_t (*afl_custom_post_process)(void *data, u8 *buf, size_t buf_size,
                                    u8 **out_buf);
  s32 (*afl_custom_init_trim)(void *data, u8 *buf, size_t buf_size);
  size_t (*afl_custom_trim)(void *data, u8 **out_buf);
  s32 (*afl_custom_post_trim)(void *data, u8 success);
  size_t (*afl_custom_havoc_mutation)(void *data, u8 *buf, size_t buf_size,
                                      u8 **out_buf, size_t max_size);
  u8 (*afl_custom_havoc_mutation_probability)(void *data);
  u8 (*afl_custom_queue_get)(void *data, const u8 *filename);
  u8 (*afl_custom_queue_new_entry)(void *data, const u8 *filename_new_queue,
                                   const u8 *filename_orig_queue);
  void (*afl_custom_deinit)(void *data);
  
  // ... and others
};

Sources: include/afl-fuzz.h:1229-1522

C/C++ API

A C/C++ custom mutator is a shared library that exports functions matching the custom_mutator struct's function pointers.

API Functions

The following table summarizes the main functions a C/C++ custom mutator can implement.

Function	Required?	Description
afl_custom_init	Yes	Initializes the mutator. Called once when the library is loaded. Returns a pointer to a custom data structure.
afl_custom_deinit	Yes	Deinitializes the mutator. Called once before afl-fuzz exits.
afl_custom_fuzz	Yes	Performs the mutation. This is the core mutation function.
afl_custom_fuzz_count	Optional	Determines how many times afl_custom_fuzz should be called for the current queue entry. Defaults to 1.
afl_custom_post_process	Optional	Modifies the mutated test case right before it is sent to the target. Should not perform random changes.
afl_custom_init_trim	Optional	Initializes the trimming process for an input. Returns the number of trimming steps.
afl_custom_trim	Optional	Performs a single trimming step.
afl_custom_post_trim	Optional	Called after a trim step to report success or failure, allowing the mutator to adjust its state.
afl_custom_havoc_mutation	Optional	A single mutation to be stacked with AFL++'s havoc stage mutations.
afl_custom_havoc_mutation_probability	Optional	Returns the probability (0-100) that afl_custom_havoc_mutation is called during the havoc stage. Default is 6%.
afl_custom_queue_get	Optional	A filter to decide whether to fuzz a given queue entry.
afl_custom_queue_new_entry	Optional	A callback invoked when a new entry is added to the queue, allowing for state updates or analysis.
afl_custom_splice_optout	Optional	If defined, this function signals that afl_custom_fuzz should not receive a splice buffer (add_buf).
afl_custom_fuzz_send	Optional	Allows the mutator to send data to the target itself, e.g., over a network socket, instead of via stdin or file.
afl_custom_describe	Optional	Provides a descriptive name for a testcase, which is useful for naming crash files.
afl_custom_introspection	Optional	Returns a string with introspection data about the mutator's state. Requires INTROSPECTION=1 build flag.

Sources: include/afl-fuzz.h:1229-1522, src/afl-fuzz-mutators.c:174-417

Python API

The Python API mirrors the C API, allowing mutators to be developed more rapidly. A Python module can define any of the functions corresponding to the C hooks.

API Functions

The functions are named identically to their C counterparts (e.g., init, fuzz, fuzz_count).

Python Function	Equivalent C Function
init(seed, afl, rand_seed)	afl_custom_init
deinit(data)	afl_custom_deinit
fuzz(data, buf, max_size)	afl_custom_fuzz
fuzz_count(data, buf)	afl_custom_fuzz_count
post_process(...)	afl_custom_post_process
init_trim(...)	afl_custom_init_trim
trim(...)	afl_custom_trim
post_trim(...)	afl_custom_post_trim
havoc_mutation(...)	afl_custom_havoc_mutation
queue_new_entry(...)	afl_custom_queue_new_entry

Sources: include/afl-fuzz.h:460-530, custom_mutators/examples/README.md

Rust API

AFL++ provides safe Rust bindings for creating custom mutators. This is the recommended approach for new mutators written in Rust due to its safety and ergonomic design.

CustomMutator Trait

The core of the Rust API is the CustomMutator trait. A developer implements this trait for their mutator struct. The export_mutator! macro is then used to generate the necessary C FFI boilerplate.

// From custom_mutators/rust/custom_mutator/src/lib.rs
pub trait CustomMutator {
    type Error: Debug;

    // Required
    fn init(seed: u32) -> Result<Self, Self::Error> where Self: Sized;
    fn fuzz<'b, 's: 'b>(
        &'s mut self,
        buffer: &'b mut [u8],
        add_buff: Option<&[u8]>,
        max_size: usize,
    ) -> Result<Option<&'b [u8]>, Self::Error>;

    // Optional with default implementations
    fn fuzz_count(&mut self, buffer: &[u8]) -> Result<u32, Self::Error> { Ok(1) }
    fn post_process<'b, 's: 'b>(
        &'s mut self,
        buffer: &'b mut [u8],
    ) -> Result<Option<&'b [u8]>, Self::Error> { Ok(Some(buffer)) }
    // ... and others
}

The library must be compiled as a cdylib to produce a .so file.

# From custom_mutators/rust/custom_mutator/Cargo.toml
[lib]
crate-type = ["cdylib"]

The afl_internals feature flag can be enabled to get access to the afl_state struct within the init function, though this is marked as unsafe.

Sources: custom_mutators/rust/custom_mutator/src/lib.rs:1-55, 393-455, custom_mutators/rust/README.md, custom_mutators/rust/custom_mutator/Cargo.toml:8-11

Provided Mutator Examples

The AFL++ repository includes several production-ready and example custom mutators. These serve as excellent starting points for developing new mutators.

Mutator	Location	Description
Examples	custom_mutators/examples/	Simple examples in C and Python showing the basic API usage.
Honggfuzz	custom_mutators/honggfuzz/	A port of the mutation engine from the Honggfuzz fuzzer.
libFuzzer	custom_mutators/libfuzzer/	A partial implementation of the libFuzzer mutation engine.
Gramatron	custom_mutators/gramatron/	A grammar-based mutator. Requires a JSON automaton file describing the grammar.
Rust Examples	custom_mutators/rust/	Examples demonstrating the Rust bindings, including structured fuzzing with lain.
ZMQ Mutator	custom_mutators/zmq/	A mutator that receives seeds from a ZMQ router, useful for complex distributed fuzzing setups.

Sources: custom_mutators/README.md, custom_mutators/examples/README.md, custom_mutators/honggfuzz/README.md, custom_mutators/gramatron/gramfuzz.c, custom_mutators/zmq/README.md

Summary

The Custom Mutators API is a cornerstone of AFL++'s extensibility. It allows fuzzing experts to move beyond generic, byte-level mutations and apply domain-specific knowledge to the mutation process. By supporting C/C++, Python, and Rust, it provides a flexible and accessible way to integrate advanced, structure-aware fuzzing techniques directly into the AFL++ engine, significantly improving fuzzing effectiveness for structured data formats.