Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• tests/backend_trait_contract.rs
• tests/e2e_cli_tests.rs
• CONTRIBUTING.md
• specs/002-implement-tdd-green/plan.md
• specs/001-create-a-comprehensive/tasks.md
• specs/006-intelligent-prompt-generation/spec.md

The testing strategy for caro is built on a foundation of Test-Driven Development (TDD) and a "test-first" philosophy. This approach is non-negotiable and ensures that all functionality is defined by a clear, testable contract before any implementation code is written. The strategy emphasizes safety, correctness, and performance through a multi-layered testing hierarchy.

The core principle is the Red-Green-Refactor cycle, where failing tests are written first (RED), followed by the minimal implementation to make them pass (GREEN), and finally, code is refactored for quality while ensuring all tests remain green. This discipline is enforced throughout the development workflow, with git commits expected to show tests being added before the corresponding implementation.

Sources: specs/002-implement-tdd-green/plan.md:70-74, 102-107, CONTRIBUTING.md:295-301

TDD Workflow

The project strictly follows the TDD cycle for all feature development. This ensures that every piece of code is written to satisfy a specific, pre-defined requirement expressed as a test.

This diagram illustrates the mandated TDD cycle.

Testing Layers

The testing strategy is composed of multiple layers, ordered from high-level API contracts down to specific unit validations. The strict order of test creation is: Contract → Integration → E2E → Unit.

This diagram shows the hierarchy of test types within the project.

Contract Tests

Contract tests are the first and most critical layer. They define the public API and expected behavior of a module before it is implemented. These tests are written during the "RED" phase of TDD and must fail before any implementation code exists. They ensure that every module adheres to its specified design.

For example, the backend_trait_contract.rs file defines the expected behavior for any backend that implements the CommandGenerator trait.

Sources: specs/002-implement-tdd-green/plan.md:143-151, CONTRIBUTING.md:377-394, specs/006-intelligent-prompt-generation/spec.md:426-431

Integration Tests

Integration tests validate end-to-end workflows and the interaction between different modules. These tests are often derived directly from user stories and quickstart scenarios to ensure that the system functions correctly as a whole. They use real dependencies like the filesystem and the async runtime to provide a realistic testing environment.

An example integration test scenario is the quickstart_command_generation test, which validates the full flow from request creation to safety validation.

// Quickstart validation workflow
#[tokio::test]
async fn quickstart_command_generation() {
    // 1. Create request
    let request = CommandRequest::new("list all files", ShellType::Bash);

    // 2. Generate with mock backend
    let backend = MockBackend::new("quickstart");
    let command = backend.generate_command(&request).await.unwrap();

    // 3. Validate safety
    let validator = SafetyValidator::new();
    let result = validator.validate(&command.command, ShellType::Bash);

    // 4. Assert safe and output
    assert_eq!(result.risk_level, RiskLevel::Safe);
    println!("Generated: {}", command.command);
}

Sources: specs/002-implement-tdd-green/plan.md:73, 158-180, specs/006-intelligent-prompt-generation/spec.md:419-425

End-to-End (E2E) CLI Tests

E2E tests exercise the compiled binary from a user's perspective, validating command-line argument parsing, stdin/stdout handling, and overall application behavior. These tests ensure that the final product works as expected in a real shell environment.

Sources: tests/e2e_cli_tests.rs, specs/002-implement-tdd-green/plan.md:72

Property-Based Tests

Property-based tests are used to validate invariants and test a wide range of inputs for edge cases that might be missed by example-based tests. The proptest crate is used for this purpose, particularly for validating the safety module and command parsing logic.

Sources: specs/002-implement-tdd-green/plan.md:30, 147, CONTRIBUTING.md:280

Performance Tests

Performance is a key non-functional requirement. Benchmarks are written to ensure the application meets specific performance targets, such as CLI startup time, safety validation speed, and command generation latency. The criterion crate is mentioned for benchmarks.

Key performance targets include:

• CLI Startup: < 100ms
• Safety Validation: < 50ms
• Mock Generation: < 2s

Sources: specs/002-implement-tdd-green/plan.md:32, 149, 217-218, specs/004-implement-ollama-and/plan.md:52-56

Test Suite Structure

The tests/ directory is organized by test type, with each file serving a specific purpose.

Test File	Description	Total Tests (Feature 002)
backend_trait_contract.rs	Validates the CommandGenerator trait API for all backends.	11
safety_validator_contract.rs	Tests the safety validation system, including patterns and cross-platform logic.	17
cli_interface_contract.rs	Validates CLI argument parsing and the main application interface.	14
integration_tests.rs	Contains end-to-end workflow tests that map to user stories.	8
property_tests.rs	Contains property-based tests for validating invariants.	10
error_handling_tests.rs	Tests error scenarios and ensures graceful failure.	11
performance_tests.rs	Contains benchmarks for performance-critical paths.	9

Sources: specs/002-implement-tdd-green/plan.md:109-117, 143-151, specs/001-create-a-comprehensive/tasks.md:32-47

Running Tests

A comprehensive set of commands is provided for developers to run tests locally. Continuous feedback during development is encouraged through tools like cargo-watch.

Common Test Commands

Command	Description
cargo test	Runs the entire test suite.
cargo test --test <test_file>	Runs all tests in a specific file (e.g., backend_trait_contract).
cargo test <test_name>	Runs a single test by its function name.
cargo test -- --nocapture	Runs tests and shows println! output.
cargo watch -x test	Runs the test suite automatically on any file change.
make check	A convenience command that runs format checks, linter, and tests.
cargo bench	Runs performance benchmarks.

Sources: CONTRIBUTING.md:316-330, specs/002-implement-tdd-green/plan.md:200-205, 307-309

Continuous Integration (CI)

The project utilizes GitHub Actions for CI to ensure code quality and cross-platform compatibility. The CI pipeline is configured to run the full test suite, linter, and format checks on every change. It also handles multi-platform builds for macOS, Linux, and Windows.

The CI process ensures that all tests pass before a pull request can be merged.

Summary

The testing strategy in caro is a cornerstone of its development process, designed to build a reliable, secure, and high-performance tool. By enforcing a strict TDD methodology, the project ensures that all code is intentional and correct by design. The layered approach, combining contract, integration, E2E, property, and performance tests, provides comprehensive coverage and confidence in every change. This rigorous, test-first culture is critical for a security-sensitive application that generates and executes shell commands.

Sources: CONTRIBUTING.md:13-18, specs/002-implement-tdd-green/plan.md:18-23