Secret Tokenization & Reversal
Relevant source files
The following files were used as context for generating this wiki page:
The Secret Tokenization & Reversal system in Honmoon provides a mechanism for identifying sensitive information—such as API keys, tokens, and Personal Identifiable Information (PII)—within agent transcripts and replacing them with non-sensitive tokens. This process is reversible, allowing the gateway to restore original values before they reach upstream services while ensuring that the agent's local environment and logs remain "clean" of secrets.
This system is a core part of the Architecture Invariant regarding "No decryption surprises," ensuring that only declared protocol facts are extracted while protecting raw sensitive data.
Sources: ARCHITECTURE.md:99-100, crates/honmoon-core/src/lib.rs:5-12
System Architecture
The tokenization system is implemented within honmoon-core and is utilized by the honmoon-cli through the hook command. It combines Tier-1 PII detection and secret detection (API keys/tokens) into a unified, reversible pipeline.
The diagram shows the bidirectional flow where raw text is converted to tokens and later reversed using cryptographic keys.
Sources: ARCHITECTURE.md:52-60, crates/honmoon-core/src/secret_tokenizer/mod.rs:1-15
Core Components
| Component | Responsibility | Source |
|---|---|---|
secret_detect | Scans text for known patterns of API keys and tokens. | ARCHITECTURE.md:52-54 |
pii | Identifies Tier-1 PII (names, emails, etc.). | ARCHITECTURE.md:52-54 |
redact | Joins detection modules into the reversible tokenizer. | ARCHITECTURE.md:52-54 |
SecretTokenizer | The primary struct managing the state of tokens and reversal maps. | crates/honmoon-core/src/secret_tokenizer/mod.rs:20-40 |
Tokenization Logic
The system uses HMAC-based token generation to ensure that tokens are deterministic within a session but undecipherable without the session key. This prevents agents from inferring the original values based on the token format.
The Tokenization Lifecycle
- Detection: The input string is passed through regex and Aho-Corasick matchers to find sensitive substrings.
- Mapping: Each unique secret is mapped to a generated token (e.g.,
HONMOON_SECRET_abc123). - State Management: The
SecretTokenizermaintains an internal map between the original value and its token.
The sequence shows how the CLI interacts with the tokenizer to process raw text.
Sources: crates/honmoon-cli/src/hook.rs:15-30, crates/honmoon-core/src/secret_tokenizer/mod.rs:45-60
Reversal and Restoration
Reversal occurs when a tokenized request needs to be sent to an upstream production system. The gateway identifies tokens in the outgoing payload and replaces them with the original secrets stored in the session context.
Cryptographic Security
To prevent token hijacking or guessing, the system employs:
- HMAC-SHA256: Used for generating deterministic tokens.
- Session Salts: Ensures tokens for the same secret differ across different agent sessions.
Sources: ARCHITECTURE.md:62-65, crates/honmoon-core/src/secret_tokenizer/mod.rs:80-95
Streaming Tokenization
For high-performance scenarios where the gateway processes large streams of data (such as log forwarding or long LLM responses), a streaming tokenizer is used. This component processes data in chunks without needing to buffer the entire payload.
StreamingTokenizer State Machine
The state machine handles partial matches that span across chunk boundaries.
Sources: crates/honmoon-core/src/secret_tokenizer/streaming/mod.rs:10-50
Claude Code Integration
The honmoon-cli hook serves as the backend for the Claude Code plugin. It uses the secret tokenizer to provide "transcript hygiene," ensuring that sensitive data captured in the agent's memory does not persist in insecure logs.
| Hook Type | Action |
|---|---|
PreToolUse | Redacts secrets before a tool is executed. |
PostToolUse | Tokenizes output from tools before returning to the agent. |
UserPromptSubmit | Scans user input for accidental secret leakage. |
Sources: ARCHITECTURE.md:67-72, crates/honmoon-cli/src/hook.rs:40-55
Conclusion
The Secret Tokenization & Reversal system is critical for Honmoon's role as a security gateway. By abstracting sensitive values into reversible tokens, it allows AI agents to function effectively in production environments while maintaining a zero-trust posture regarding the handling of credentials and PII within the agent's operational transcript. Reimposing the original secrets only at the network egress point ensures that security does not come at the cost of functionality.
Sources: ARCHITECTURE.md:82-100, crates/honmoon-core/src/lib.rs:1-12