Automatic Dictionary Generation
Relevant source files
The following files were used as context for generating this wiki page:Automatic dictionary generation in AFL++ is a set of features designed to automatically extract tokens, keywords, and magic numbers from a target's source code or runtime behavior. These generated dictionaries, when used with afl-fuzz -x, significantly improve the fuzzer's ability to explore complex, grammar-based data formats by providing it with meaningful syntax units. This leads to better code coverage and a higher chance of discovering new bugs, especially in parsers.
AFL++ offers several distinct mechanisms for automatic dictionary generation, each with its own approach. These include static analysis with CodeQL (Autodict-QL), compile-time extraction during LTO (afl-clang-lto), and a token-level custom mutator (autotokens). These tools provide automated, target-specific dictionary creation, which is often more effective than using pre-built, generic dictionaries.
Dictionary Usage in afl-fuzz
Regardless of the generation method, the resulting dictionaries are consumed by afl-fuzz to guide its mutation strategies.
Command-Line Options
-x dict_file: Specifies a dictionary file or a directory of token files to be used during fuzzing. This option can be used up to four times.-z: Skips the enhanced deterministic fuzzing stages, which heavily utilize dictionary tokens.
Sources: src/afl-fuzz.c:266-267, 270-271, 1005-1012, 1142-1144
Dictionary Formats
AFL++ supports two formats for custom dictionaries:
- Directory Mode: A directory containing individual files, where each file's content is a single token.
- File Mode: A flat text file where tokens are listed one per line in the format
name="value". Thenameis ignored. Thevaluemust be in quotes, with non-printable characters hex-escaped (e.g.,\xNN).
The fuzzer automatically detects the mode based on whether the path given to -x is a file or a directory.
Sources: dictionaries/README.md:10-21
Important Environment Variables
AFL_MAX_DET_EXTRAS: If the number of tokens in the dictionary exceeds this value (default: 200), the fuzzer will only use a random subset probabilistically. To ensure all generated tokens are used in deterministic stages, this variable should be set to at least the number of tokens in the dictionary.AFL_NO_AUTODICT: If set, this variable preventsafl-fuzzfrom loading an automatically generated dictionary that may have been compiled into the target binary (e.g., byafl-clang-lto).
Sources: utils/autodict_ql/readme.md:118-122, src/afl-fuzz.c:560-567, instrumentation/README.lto.md:101-103
Autodict-QL
Autodict-QL is a scriptable system that uses the CodeQL semantic code analysis engine to extract tokens from a target's source code. It is independent of the LLVM toolchain and allows users to write or modify queries to customize token extraction logic.
Sources: utils/autodict_ql/readme.md:5-12
Architecture and Workflow
The process involves building a CodeQL database from the target's source, running a series of queries against it, and processing the results into a dictionary folder.
The following diagram illustrates the Autodict-QL workflow:
CodeQL Scripts
Autodict-QL includes several default scripts for extracting common token types:
| Script | Description |
|---|---|
strcmp-str.ql | Extracts string arguments from strcmp function calls. |
strncmp-str.ql | Extracts string arguments from strncmp function calls. |
memcmp-str.ql | Extracts string arguments from memcmp function calls. |
litool.ql | Extracts magic numbers and other literal values in hexadecimal format. |
strtool.ql | Uses dataflow analysis to find string comparison functions (including custom wrappers) and their arguments. |
Sources: utils/autodict_ql/readme.md:21-34
Usage
The main script is autodict-ql.py, which orchestrates the analysis.
Arguments:
| Argument | Description |
|---|---|
cur | The full path to the current directory. |
db | The full path to the CodeQL database. |
tokenpath | The name of the output folder for tokens. |
Sources: utils/autodict_ql/autodict-ql.py:4-11, 26-33
The script executes a series of codeql query run commands and then processes the output with helper Python scripts (litan.py, strcmp-strings.py, etc.) to generate the final token files.
Sources: utils/autodict_ql/autodict-ql.py:80-83, 91-115
LTO-mode AUTODICTIONARY
When compiling a target with afl-clang-lto, an automatic dictionary generation feature is enabled by default.
How It Works
- Compile-Time Extraction: During Link-Time Optimization (LTO),
afl-clang-ltoanalyzes the code for string comparisons. - Binary Embedding: The extracted tokens are embedded directly into the compiled target binary.
- Fuzzer Loading: When
afl-fuzzstarts, it detects these embedded tokens and automatically loads them into its dictionary, as if they were provided with-x.
This provides a seamless way to generate a dictionary without extra steps, improving coverage by 5-10% on average. To disable this feature, set the AFL_NO_AUTODICT=1 environment variable when running afl-fuzz.
Sources: instrumentation/README.lto.md:97-103, src/afl-fuzz.c:4298-4301
autotokens Custom Mutator
The autotokens custom mutator is a grammar-agnostic, token-level fuzzer designed for ASCII-based formats. Instead of generating a static dictionary file, it tokenizes inputs on-the-fly and performs mutations at the token level.
Tokenization Process
The mutator parses an input into a sequence of tokens based on a set of rules:
- Strings: Quoted strings (
"or') are treated as single tokens. - Comments: C-style block comments (
/* ... */) are removed. A custom comment regex can be supplied via theAUTOTOKENS_COMMENTenvironment variable. - Whitespace: Sequences of spaces and tabs are identified as whitespace tokens.
- Words: Alphanumeric identifiers (plus
_,$,.,-) are treated as word tokens. - Other: Any remaining characters are treated as single-character tokens.
Each unique token is assigned an ID, and the input file is converted into a structure of token IDs.
Sources: custom_mutators/autotokens/autotokens.cpp:408-547, 694-700
Mutation Strategy
Mutations are performed on the token ID structure, not the raw bytes. The primary mutation operations include:
- Change/Mutate: A single token is replaced with another randomly selected token.
- Insert: A new token is inserted at a random position.
- Splice: A sequence of tokens from another parsed input is inserted or used to overwrite a section of the current structure.
- Erase: A single token is removed from the structure.
After mutation, the token ID structure is reassembled into a new input file. The mutator ensures that a whitespace token is inserted between two consecutive word/string tokens to maintain basic syntax validity.
Sources: custom_mutators/autotokens/autotokens.cpp:208-356
Comparison of Methods
| Method | Type | How it Works | When to Use |
|---|---|---|---|
| Autodict-QL | Static Analysis | Runs CodeQL queries on source to find strings, magic numbers. | When you have the source code and want a scriptable, LLVM-independent way to generate a comprehensive dictionary. |
| LTO AUTODICTIONARY | Compile-Time | Extracts string comparison tokens during LTO compilation and embeds them. | When using afl-clang-lto for a simple, integrated dictionary generation process. |
autotokens | Custom Mutator | Tokenizes ASCII inputs on-the-fly and mutates at the token level. | For fuzzing text-based formats where grammar is defined by whitespace and identifiers, without needing a formal grammar definition. |
Sources: utils/autodict_ql/readme.md, instrumentation/README.lto.md, custom_mutators/autotokens/autotokens.cpp
Conclusion
AFL++ provides a powerful and flexible suite of tools for automatic dictionary generation. By leveraging static analysis with Autodict-QL, integrated compile-time extraction with LTO AUTODICTIONARY, or dynamic token-level mutation with autotokens, users can create highly effective, target-specific dictionaries. Using these features is highly recommended over manual dictionary creation or generic dictionaries, as they significantly enhance the fuzzer's ability to navigate the logic of complex parsers, leading to deeper code coverage and more discovered vulnerabilities. For best results, combining automatically generated dictionaries with CmpLog-guided fuzzing is advised.