Testing Strategy & Harness Integration
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The testing strategy for Continuous Claude-v2 centers on a multi-layered verification approach encompassing unit tests, integration tests, and specialized harness execution. This framework ensures the reliability of core components such as the Model Context Protocol (MCP) client, artifact indexing, and cross-session state management. By utilizing a dedicated runtime harness, the system can execute complex Python workflows with pre-initialized MCP connections, reducing overhead and improving execution consistency.
The strategy emphasizes "TDD" (Test-Driven Development) for critical path features like the Context Graph query interface and artifact schema validation. It also incorporates automated performance and quality analysis via the braintrust_analyze.py script, which evaluates session metrics, detects tool loops, and facilitates LLM-as-judge critiques for implementation plans.
Sources: tests/test_artifact_query.py:7-10, scripts/braintrust_analyze.py:1-40, tests/integration/test_harness_integration.py:1-5
Execution Harness Architecture
The runtime.harness serves as the primary execution environment for MCP-orchestrated workflows. It manages the lifecycle of scripts, including asynchronous initialization of the MCP Client Manager, signal handling, and final cleanup. This harness allows scripts to call tools using a unified tool ID format (serverName__toolName) without manually managing server connections.
Harness Workflow
The harness follows a strict lifecycle for script execution:
- Detection: Identifying the correct Python executable (prioritizing
.venv/bin/python). - Initialization: The harness pre-initializes the
McpClientManager. - Execution: The target script is run within the initialized context, enabling direct calls to
call_mcp_tool. - Validation: Capture of
stdoutandstderrto verify successful execution or handle exceptions.
The harness ensures that scripts can interact with the underlying Git or MCP infrastructure seamlessly.
Sources: tests/integration/test_harness_integration.py:8-66, CLAUDE.md:65-72
Artifact Indexing & Schema Validation
A core part of the testing strategy involves validating the "Context Graph"—a SQLite-based storage system for handoffs, plans, and continuity ledgers. Testing ensures that the SQL schema is valid, FTS5 virtual tables are correctly mapped for full-text search, and triggers automatically synchronize content between main tables and search indexes.
Database Components
| Component | Purpose | Verified In |
|---|---|---|
| Main Tables | Persistence for handoffs, plans, continuity, and queries. | test_artifact_schema.py |
| FTS5 Tables | Full-text search capability using BM25 ranking. | test_artifact_query.py |
| Triggers | Automatic after insert, after delete, and after update sync. | test_artifact_schema.py |
| Constraints | Outcome mapping (e.g., SUCCEEDED, FAILED) and snapshot reason validation. | test_artifact_schema.py |
Tool Discovery and Safety Testing
The system includes automated classification of MCP tools to determine if they are safe for discovery or potentially dangerous. This is critical for generating test parameters without causing unintended side effects (e.g., deleting data).
Safety Classification Logic
Tools are classified based on naming patterns and description keywords:
- SAFE: Prefixes like
get_,list_,search_,read_, or keywords like "Fetch", "View". - DANGEROUS: Prefixes like
delete_,remove_,drop_,write_,execute_, or keywords like "Destroy", "Modify". - UNKNOWN: Tools without clear patterns require manual review or specific mock overrides.
Sources: tests/unit/test_generate_test_params.py:16-55, src/runtime/generate_test_params.py
Automated Discovery Workflow
Analysis and Quality Metrics
The testing strategy extends into post-execution analysis via the braintrust_analyze.py script. This tool evaluates session quality through both quantitative metrics and qualitative "LLM-as-judge" critiques.
Key Analysis Features
- Loop Detection: Identifies sessions where a tool is called more than 5 times consecutively, signaling a logic error.
- Token Trends: Tracks K-token consumption per session to monitor context growth.
- RAG-Enhanced Judging: Uses Context Graph precedents (past successes and failures) to critique implementation plans before they are executed.
- Implementation Review: Compares a Plan (intent) against Git Diffs (reality) and Session Summaries to identify P0 gaps.
Sources: scripts/braintrust_analyze.py:270-300, scripts/braintrust_analyze.py:535-580, scripts/braintrust_analyze.py:615-660
Conclusion
The testing strategy for Continuous Claude-v2 is a robust framework that combines traditional unit/integration testing with modern AI-driven analysis. By leveraging a dedicated execution harness and a structured Context Graph, the project ensures that state preservation and tool orchestration are verified at every step. The integration of LLM-based implementation reviews further shifts testing "left" by validating plans against historical precedents before code is even written.
Sources: scripts/braintrust_analyze.py:870-900, tests/test_artifact_query.py:10-15