Adding a New Safety Pattern

Relevant source files

The following files were used as context for generating this wiki page:

• src/safety/patterns.rs
• CONTRIBUTING.md
• tests/safety_validator_contract.rs
• specs/002-implement-tdd-green/plan.md
• specs/002-implement-tdd-green/research.md
• presentation/slides.md
• CONTRIBUTORS.md
• exports/session-logs/2025-10-01-check-the-pr.txt

The caro project is a security-critical tool that translates natural language into executable shell commands. A core tenet of the project is "Safety First," and the primary mechanism for ensuring safety is a robust validation system that checks generated commands against a database of dangerous patterns. This system is designed to be the final gatekeeper before any command is executed, providing defense-in-depth against potentially harmful LLM outputs.

Contributing new safety patterns is one of the most impactful ways to improve the project. By adding patterns based on real-world experience, contributors help build a collective knowledge base that protects all users from destructive commands. This document outlines the architecture of the safety system and the process for adding new patterns.

Sources: CONTRIBUTING.md:10-14, HANDBOOK.md:19-25, presentation/slides.md:81-89

Safety System Architecture

The safety system is built around the SafetyValidator module. Its responsibility is to analyze a generated command string and assess its potential risk before execution. This validation is performed locally and does not depend on any network services, ensuring it works offline and is fast.

Sources: specs/002-implement-tdd-green/spec.md:31-38, specs/002-implement-tdd-green/plan.md:200-205

Validation Flow

The validation process follows a clear sequence. When a command is generated, it is passed to the SafetyValidator, which matches it against the pattern database. Based on the matched pattern's severity, a RiskLevel is assigned, which determines the final action.

The diagram below illustrates the command validation flow.

Risk Levels

The system uses a four-tier risk assessment model. Each safety pattern is associated with a specific RiskLevel.

Risk Level	Action	Description
Critical	Blocked	Destructive operations that are always blocked, such as rm -rf / or mkfs.
High	Confirmation Required	Potentially dangerous operations that require explicit user approval, like rm -rf on other paths or package installations.
Moderate	Warning	Operations that are generally safe but can have unintended consequences, such as file moves or large searches. Confirmation is required in Strict mode.
Safe	Allowed	Read-only operations and other commands with no perceived risk.

Sources: presentation/slides.md:92-111, specs/002-implement-tdd-green/plan.md:112-122, CLAUDE_SKILL_ISSUE.md:101-118

Defining a Safety Pattern

A safety pattern consists of a regular expression, a risk level, and a human-readable description. These are defined in the DangerPattern struct.

The diagram below shows the structure of a safety pattern.

Pattern Implementation

Core safety patterns are stored in src/safety/patterns.rs. For performance, all regex patterns are compiled once at first use and stored in a static variable using once_cell::sync::Lazy. This avoids expensive re-compilation on every validation call, ensuring the safety check is extremely fast.

Sources: specs/002-implement-tdd-green/plan.md:71-76, specs/002-implement-tdd-green/research.md:65-85

Here is an example of how patterns are defined:

// Source: specs/002-implement-tdd-green/research.md:70-85

use once_cell::sync::Lazy;
use regex::Regex;

static DANGEROUS_PATTERNS: Lazy<Vec<(Regex, &'static str, RiskLevel)>> = Lazy::new(|| {
    vec![
        (
            Regex::new(r"rm\s+-rf\s+/").unwrap(),
            "Recursive deletion of root filesystem",
            RiskLevel::Critical
        ),
        (
            Regex::new(r":\(\)\{.*:\|:&\s*\};:").unwrap(),
            "Fork bomb detected",
            RiskLevel::Critical
        ),
        // ... more patterns
    ]
});

This approach ensures that the first validation call may be slightly slower, but all subsequent calls are highly optimized, meeting the performance target of <50ms per validation.

Sources: specs/002-implement-tdd-green/research.md:87-92, specs/002-implement-tdd-green/spec.md:66-67

Contribution Workflow

Adding a new safety pattern follows a strict Test-Driven Development (TDD) workflow. Contributors must first write a failing test that proves the need for the pattern.

The diagram below shows the contribution workflow.

Step 1: Propose a New Pattern

Before writing code, propose the new pattern by opening an issue on GitHub. Use the "Safety Pattern Submission" template to describe the command, explain why it's dangerous, and suggest a risk level. This allows for community discussion before implementation.

Sources: CONTRIBUTORS.md:30-34, CONTRIBUTING.md:477-479

Step 2: Write a Failing Test

The project enforces a TDD methodology. You must add a new test to tests/safety_validator_contract.rs that validates your new pattern. The test should initially fail because the pattern does not yet exist in the validator.

The test_custom_safety_patterns test serves as a great example for how to structure this.

// Source: exports/session-logs/2025-10-01-check-the-pr.txt:631-676

#[tokio::test]
async fn test_custom_safety_patterns() {
    // CONTRACT: Should support custom safety patterns for project-specific rules
    let mut config = SafetyConfig::moderate();

    // Add custom pattern for blocking specific commands
    config.add_custom_pattern(DangerPattern {
        pattern: r"git\s+push\s+--force".to_string(),
        risk_level: RiskLevel::High,
        description: "Force push can overwrite remote history".to_string(),
        shell_specific: None,
    });

    let validator = SafetyValidator::new(config).unwrap();

    // Test that custom pattern is detected
    let force_push = "git push --force origin main";
    let result = validator
        .validate_command(force_push, ShellType::Bash)
        .await
        .unwrap();

    assert!(
        matches!(result.risk_level, RiskLevel::High),
        "Custom pattern should flag force push as high risk"
    );
    
    // Regular git push should still be allowed
    let regular_push = "git push origin main";
    let safe_result = validator
        .validate_command(regular_push, ShellType::Bash)
        .await
        .unwrap();

    assert!(
        safe_result.allowed,
        "Regular git push should be allowed with moderate safety"
    );
}

This test demonstrates how to add a custom pattern and assert that it is correctly identified while non-matching commands are still allowed.

Sources: exports/session-logs/2025-10-01-check-the-pr.txt:631-676

Step 3: Add the Pattern

Once you have a failing test, add your new pattern to the DANGEROUS_PATTERNS static vector in src/safety/patterns.rs. Ensure you provide an accurate regex, the correct RiskLevel, and a clear description.

Sources: specs/002-implement-tdd-green/plan.md:190-192

Step 4: Submit a Pull Request

After your test passes, run the full validation suite (make check) to ensure no regressions were introduced. Then, submit a pull request linking to the issue you created in Step 1. Your PR will be reviewed by maintainers.

Sources: CONTRIBUTING.md:491-522

Conclusion

The safety pattern database is a cornerstone of caro's commitment to user safety. By contributing new patterns, developers directly enhance the tool's ability to prevent accidents and build a trusted, collective defense against dangerous commands. The TDD-based workflow ensures that every pattern is verified, documented, and robust before it is merged.

Sources: HANDBOOK.md:19-25, CONTRIBUTING.md:472-476