Safety & Validation Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Safety & Validation Pipeline is a foundational component of caro, designed to ensure that shell commands generated by Large Language Models (LLMs) are safe to execute. As a security-critical tool, caro prioritizes user safety by implementing a robust, multi-layered validation process that operates independently of the LLM's own assessment. This is based on the critical finding that models can misclassify dangerous commands (e.g., marking rm -rf / as "Safe"). The pipeline's primary function is to parse, analyze, and risk-assess every generated command against a comprehensive set of patterns and rules before presenting it to the user for execution.
This system is designed with a "defense in depth" philosophy, never trusting the LLM output and enforcing secure defaults. It is a non-negotiable part of the command generation flow, ensuring that caro is a trustworthy assistant for terminal operations.
Sources: CONTRIBUTING.md:12-18, mlx-test/DELIVERABLES.md:40-43, presentation/slides.md:108-119
Architecture Overview
The Safety & Validation Pipeline is an integral part of the command generation workflow. After a command is generated by an LLM backend, it is immediately passed to the SafetyValidator for analysis. The validator assesses the command for potential risks, checks for platform-specific incompatibilities, and assigns a risk level. Based on this assessment, the command may be blocked, require user confirmation, or be approved for execution.
The following diagram illustrates the high-level data flow through the validation pipeline.
This diagram shows the flow of a command from generation through validation, risk assessment, and final action based on the determined risk level.
Sources: specs/001-create-a-comprehensive/tasks.md:35-36, presentation/slides.md:83-98, CONTRIBUTING.md:150-161
Core Components
The pipeline is composed of several key data structures and modules that work together to provide a comprehensive safety net.
Data Structures
The core data structures define the inputs, outputs, and configuration of the validation system.
This diagram shows the relationships between the main components: SafetyValidator uses a SafetyConfig and produces a ValidationResult, which contains a RiskLevel.
Sources: specs/002-implement-tdd-green/plan.md:120-155, specs/006-intelligent-prompt-generation/spec.md:364-386
Safety Configuration
The behavior of the SafetyValidator is controlled by the SafetyLevel enum, which allows users to choose their desired level of protection.
| Level | Description |
|---|---|
Strict | Blocks any command rated High or Critical. Requires user confirmation for Moderate risk commands. |
Moderate | (Default) Blocks Critical commands. Requires confirmation for High risk commands. Warns on Moderate. |
Permissive | Warns on all risk levels but does not block or require confirmation. Intended for advanced users. |
Sources: specs/002-implement-tdd-green/plan.md:144-149
Risk Assessment
Risk assessment is the core of the pipeline, classifying commands into one of four levels. This is done by matching the command against a database of pre-compiled regular expressions.
| Risk Level | Action | Example Patterns |
|---|---|---|
Critical | Blocked by default. | rm -rf /, mkfs.*, `:(){ : |
High | Requires explicit user confirmation. | rm -rf, chmod 777, sudo su |
Moderate | Warns the user before execution. | File copying/moving, archive operations, permission changes. |
Safe | Allowed to proceed without interruption. | Read-only operations like ls, cat, grep. |
This tiered system ensures that the most destructive commands are always blocked, while potentially risky ones are brought to the user's attention.
Sources: presentation/slides.md:99-111, CONTRIBUTING.md:374-385, exports/session-logs/2025-09-13-readme.txt:100-104
Validation Logic
Dangerous Pattern Matching
The primary method for risk assessment is matching commands against a database of known dangerous patterns, defined in src/safety/patterns.rs. The system uses the regex crate with once_cell::sync::Lazy to compile these patterns once at first use for optimal performance. The validation is designed to be entirely local and network-independent.
Sources: specs/001-create-a-comprehensive/tasks.md:36, specs/002-implement-tdd-green/plan.md:213-218, tests/safety_validator_contract.rs:631-641
Cross-Platform Validation
The validator is designed to be platform-aware, handling differences between POSIX-compliant shells (like bash and zsh on GNU/Linux) and BSD-derivatives (like on macOS). This prevents failures from generating commands with incompatible flags, such as using GNU's --sort=size on a BSD system that expects -S.
The planned architecture includes a ManPageAnalyzer that can parse man pages to build a cache of available flags for tools on the specific system, ensuring hyper-accurate validation.
The system also supports generating commands for a target platform that is different from the host, with warnings to the user.
Sources: specs/006-intelligent-prompt-generation/spec.md:78-92, 193-219, tests/safety_validator_contract.rs:502-590
Multi-Agent Validation Flow
A more advanced, multi-turn validation flow is specified to handle cases where initial command generation fails. This involves a coordination of multiple agents.
This sequence shows how the system can self-correct an invalid command by using feedback from the Validator to inform the Generator on a subsequent attempt.
Sources: specs/006-intelligent-prompt-generation/spec.md:506-532
Testing
The safety pipeline is rigorously tested through a series of contract tests defined in tests/safety_validator_contract.rs. These tests form a contract that the implementation must fulfill.
Key test categories include:
- Dangerous Command Detection: Ensures patterns for
rm -rf,mkfs, fork bombs, etc., are correctly identified. - Risk Level Assessment: Verifies that commands are assigned the correct
RiskLevel. - Cross-Platform Path Handling: Tests validation of both POSIX and Windows style paths.
- Custom Pattern Support: Confirms that users can add their own project-specific safety rules.
- Network Resilience: Guarantees that validation works entirely offline.
- Performance: Benchmarks validation speed to be under 50ms.
Sources: tests/safety_validator_contract.rs, specs/002-implement-tdd-green/spec.md:27-42
Conclusion
The Safety & Validation Pipeline is the cornerstone of caro's commitment to user trust and security. By explicitly distrusting LLM outputs and implementing an independent, multi-layered, and locally-run validation system, caro provides a powerful yet safe interface to the command line. The architecture is designed to be both robust and extensible, with plans for advanced features like a ManPageAnalyzer and community-governed pattern databases, ensuring the tool remains safe and intelligent as it evolves.
Sources: CONTRIBUTING.md:19-24, mlx-test/DELIVERABLES.md:99-102