Gitleaks Integration

Relevant source files

The following files were used as context for generating this wiki page:

• config/gitleaks.toml
• config/oboete-rules.toml
• src/redact.rs
• src/capture.rs
• src/config.rs

Oboete integrates Gitleaks rules to mask secrets before text is stored or transmitted. The system applies redaction during the capture of agent interactions and again when data leaves the machine. This ensures that sensitive credentials, such as API keys and authorization tokens, do not persist in long-term memory or reach external providers.

The integration uses the Gitleaks configuration schema to define scanning rules. It combines a bundled set of default rules with project-specific additions and user-defined rules. The redaction logic identifies secret patterns using regular expressions gated by keywords to optimize performance.

Sources: src/redact.rs:1-10, src/capture.rs:1-8

Redaction Architecture

The redaction system operates as a gate for all text data. It processes strings by identifying patterns defined in TOML configuration files and replacing them with a standard mask.

Component Overview

The redaction system consists of three main rule layers:

1. Bundled Gitleaks Rules: A standard set of secret scanning rules provided by the Gitleaks project.
2. Oboete Rules: Project-specific rules for common agent provider keys and specific header formats.
3. User Rules: Custom rules defined by the user in the project's config.toml.

Sources: src/redact.rs:14-20, src/config.rs:596-610

Redaction Logic Flow

When text enters the system, Oboete applies the following sequence:

The system performs multiple passes (rescans) to ensure that secrets requiring context are caught even after surrounding text is masked. For example, a curl command might have its authorization header masked in one pass, potentially revealing or altering the context needed for a second rule.

Sources: src/redact.rs:243-260, src/redact.rs:320-350

Rule Definition and Configuration

Oboete uses TOML files to define how secrets are identified. Each rule requires a unique ID and a regular expression.

Rule Structure

Field	Description
id	Unique identifier for the rule used in logs and ledgers.
regex	The pattern used to identify the secret.
keywords	Case-insensitive strings that must be present to trigger the regex.
entropy	Minimum Shannon entropy threshold to filter out low-randomness strings.
secretGroup	The capture group index containing the actual secret.

Sources: src/redact.rs:27-46, config/oboete-rules.toml:1-10

Default Oboete Rules

Oboete includes specific rules for AI providers and authorization headers that are not part of the standard Gitleaks set.

[[rules]]
id = "groq-api-key"
description = "Groq API key"
regex = '''\b(gsk_[A-Za-z0-9]{40,64})\b'''
keywords = ["gsk_"]

[[rules]]
id = "authorization-header"
description = "Credential in an Authorization header"
regex = '''(?i)\b(?:proxy-)?authorization["']?\s*[:=]\s*["']?...'''
keywords = ["authorization"]
entropy = 3.0

Sources: config/oboete-rules.toml:5-10, config/oboete-rules.toml:53-58

Performance Optimization

Scanning large volumes of text for hundreds of regular expressions is computationally expensive. Oboete employs several strategies to maintain high performance during agent interactions.

Keyword Gating

The system uses an Aho-Corasick automaton to perform a single case-insensitive pass over the text to find keywords. A regular expression for a specific rule is only compiled and executed if one of its associated keywords is detected in the input string.

Sources: src/redact.rs:88-105, src/redact.rs:204-215

Regex Compilation and Caching

Regular expressions are compiled on-demand and cached in thread-local storage for the duration of the process. Oboete prefers ASCII mode for compilation, which is significantly faster than Unicode mode for the generic patterns found in secret scanning.

Sources: src/redact.rs:188-202

Data Capture and Capping

The Gitleaks integration is invoked by capture.rs whenever a hook payload is processed.

The Redaction Gate

Oboete uses a Gate structure during capture to track every string and key in a JSON payload. It applies redact::scan_capped to ensure that even truncated strings are fully scanned before being stored in the database.

Field Capping

If a string exceeds MAX_FIELD_BYTES (64 KB), Oboete keeps only the head and tail. Redaction is applied to the entire string before this cut occurs. This prevents a secret from being leaked because it was split across the truncation point.

Sources: src/capture.rs:18-20, src/redact.rs:372-390

Redaction Ledger

For every secret masked during capture, Oboete stores a finding in a ledger. This ledger records the rule ID, the offset within the masked text, and the original length of the secret. This metadata allows the system to understand what was masked without ever storing the sensitive value itself.

Sources: src/capture.rs:75-80, src/redact.rs:275-285

Conclusion

The Gitleaks integration provides a robust security layer for Oboete. By leveraging optimized keyword gating and multi-pass scanning, it identifies secrets in both raw text and JSON-escaped formats. The system's design ensures that redaction rules are applied consistently during initial data ingestion and whenever data is retrieved for external use.

Sources: src/redact.rs:1-5, src/capture.rs:1-5