Data Privacy & Redaction

Relevant source files

The following files were used as context for generating this wiki page:

• src/redact.rs
• config/gitleaks.toml
• src/curate.rs
• src/manifest.rs
• CLAUDE.md

The Data Privacy & Redaction system protects sensitive information by masking secrets before they are stored or transmitted. It ensures that API keys, tokens, and private data remain isolated from external providers and logs. The system applies a combination of automated scanning rules, user-defined configurations, and content-aware fencing to maintain data integrity and owner privacy.

Redaction Architecture

The redaction system processes text through a multi-pass scanning engine. It uses keywords to gate computationally expensive regex matches, identifies secrets, and replaces them with a uniform mask.

Scanning Logic

Redaction involves several stages to ensure thorough coverage:

1. Keyword Gating: The system uses the Aho-Corasick algorithm to scan for keywords. A regex only runs if a corresponding keyword hits.
2. Regex Matching: It compiles regex patterns to identify specific secret shapes (e.g., AWS keys, GitHub tokens).
3. Entropy Filtering: Candidates must pass Shannon entropy checks to distinguish real secrets from random strings or common prose.
4. Allowlisting: The system ignores matches that appear in global or rule-specific allowlists (e.g., documented example keys).
5. Multi-pass Rescanning: It scans masked text repeatedly (up to 64 passes) until no new secrets are found. This handles cases where context for one secret is revealed after masking another.

Sources: src/redact.rs:1-20, src/redact.rs:445-480, src/redact.rs:518-530

Multi-Pass Redaction Flow

This diagram shows how text is processed to reaching a redacted fixpoint.

Redaction Rules and Configuration

The system uses both bundled rules and user-provided configurations. Rules are stored in TOML format and define the identifiers for various secret types.

Rule Components

A redaction rule consists of several fields:

Field	Description
id	Unique identifier for the rule (e.g., aws-access-token).
regex	The pattern used to capture the secret.
keywords	String tokens that trigger the regex scan.
entropy	Minimum Shannon entropy required for a match to be redacted.
allowlists	Patterns or stopwords that exempt a match from redaction.

Sources: src/redact.rs:27-45, config/gitleaks.toml:130-145

User-Defined Rules

Users can extend the system via the [redaction] table in config.toml.

• Extra Rules: Custom regexes and keywords prefixed with user:.
• Allowlist: SHA-256 hashes of specific values the user wishes to keep (e.g., non-sensitive internal IDs that look like tokens).

Sources: src/redact.rs:85-110, src/redact.rs:150-165

Content-Aware Privacy

Beyond simple string matching, the project implements context-aware privacy measures for specific data structures.

Block Removal

The system removes closed <private>-style tags before scanning. This handles blocks like <private>...</private> or memory context tags.
Sources: src/redact.rs:320-335

JSON Unescaping

Since tools often return flattened JSON, the system performs a scan on an unescaped view of the text. This ensures that secrets hidden behind JSON escape sequences (like \" or \n) are still identified.
Sources: src/redact.rs:595-615

Manifest Isolation

When generating a session manifest, the system fences data to prevent models from interpreting it as instructions. It also filters "Owner's directives" by checking for negation markers (e.g., "never mind") that cancel out previous instructions.
Sources: src/manifest.rs:55-70, src/manifest.rs:135-145

Redaction Limits and Constraints

To prevent performance degradation or unbounded data growth, the system enforces several caps:

• Max Findings: If a text contains more than 1,000 findings, the entire text is replaced with a single mask.
• Max Looks: A rule is limited to 10,000 matches per text to avoid catastrophic backtracking or excessive processing of repetitive patterns.
• Op Cap: Window operations are cut to fit within the MAX_OP_BYTES limit, dropping metadata about redacted IDs if necessary to ensure the record can be stored.

Sources: src/redact.rs:485-495, src/curate.rs:885-900

Curation Gate Integration

During the curation phase, text is passed through the "egress gate."

Component	Privacy Action
Tool Input	Cut at a character limit (e.g., 2,000 chars) before redaction.
Tool Output	Long outputs are shrunk to head and tail, with the middle elided.
Memory Reads	Output from tools reading internal memory (e.g., oboete search) is replaced with a placeholder to prevent recursive loops.
Evidence	Quotes used as evidence must be found verbatim in redacted text; quotes containing masks are rejected.

Sources: src/curate.rs:360-380, src/curate.rs:430-450, src/curate.rs:695-710

Summary

The Data Privacy & Redaction system provides a robust defense against accidental data exposure. By combining strict Gitleaks-style pattern matching with context-sensitive logic—such as JSON unescaping and instruction fencing—it ensures that sensitive information is stripped at every stage of the data lifecycle. The multi-pass scanning architecture ensures that even complex, interleaved secrets are consistently masked before leaving the local environment.