Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• TESTING.md
• CONTRIBUTING.md
• CLAUDE.md
• README.md
• tests/helpers/fixtures.bash
• create_files.sh
• templates/AGENT.md

The testing strategy for the Ralph project is a comprehensive system designed to ensure code quality, prevent regressions, and validate the behavior of its autonomous development loop. The strategy relies on the Bash Automated Testing System (BATS) and is a mandatory part of the development workflow, with a strict 100% test pass rate enforced by the CI/CD pipeline. The project currently has 276 tests, all passing.

The primary goal is to validate behavior rather than just achieving high coverage metrics. All new features must include corresponding tests, and contributions are not accepted without them. This rigorous approach is critical for maintaining the stability and reliability of the autonomous agent.

Sources: TESTING.md:3-5, README.md:8, 304, CONTRIBUTING.md:120-123, CLAUDE.md:501-507

Core Principles and Quality Gates

The testing philosophy prioritizes correctness and stability. While an 85% code coverage target is aspirational, the primary quality gate is a 100% test pass rate. This is because the coverage tool, kcov, has technical limitations in tracing subprocesses spawned by BATS, making the pass rate a more reliable metric.

Sources: TESTING.md:313-324, CONTRIBUTING.md:125-131, CLAUDE.md:143-149

Quality Gates

All pull requests must pass these automated checks before being merged.

Gate	Requirement	Enforcement
Unit Tests	100% pass	Blocks merge
Integration Tests	100% pass	Blocks merge
Test Coverage	85% (Aspirational)	Informational only
Conventional Commits	Required	Manual review
Documentation	Must be updated	Manual review

Sources: CONTRIBUTING.md:209-217, templates/AGENT.md:22-26

Feature Completion Checklist

For any new feature to be considered complete, it must meet stringent testing requirements.

• All tests must pass with a 100% pass rate.
• New code must have a minimum of 85% code coverage.
• Required test types include Unit, Integration, and End-to-end tests for critical workflows.
• All changes must be committed with conventional commit messages and pushed to the remote repository.

Sources: templates/AGENT.md:20-39, 90-107, CLAUDE.md:501-510

Test Organization and Structure

The test suite is organized into distinct categories within the tests/ directory, ensuring a clear separation of concerns and allowing developers to run specific test types as needed.

Sources: TESTING.md:61-79, CONTRIBUTING.md:65-79

The diagram below illustrates the test directory structure.

Test Categories

Category	Purpose	Speed	Dependencies
Unit	Test individual functions in isolation.	Fast (<1s per file)	None (uses mocks)
Integration	Test interactions between multiple components.	Medium (1-5s per file)	Real git, filesystem
E2E	Test complete end-to-end user workflows.	Slow (>5s per file)	Full environment

Sources: TESTING.md:81-86, CONTRIBUTING.md:134-136

Test Framework and Execution

The project uses the Bash Automated Testing System (BATS) as its core testing framework, along with helper libraries bats-assert and bats-support. Dependencies are managed via npm.

Sources: TESTING.md:23-27, CONTRIBUTING.md:32-34

Running Tests

A set of npm scripts are provided to execute the test suite.

Command	Purpose
npm test	Run the complete test suite (unit + integration).
npm run test:unit	Run only the fast, isolated unit tests.
npm run test:integration	Run only tests for component interactions.
npm run test:e2e	Run end-to-end workflow tests.
bats tests/unit/test_file.bats	Run a single test file for debugging.

Sources: TESTING.md:33-51, CONTRIBUTING.md:150-155

Test Structure Example

All tests follow a standard structure with setup() and teardown() functions to ensure test isolation. The run command is used to execute a function and capture its status and output for assertion.

#!/usr/bin/env bats
# Unit Tests for Rate Limiting Logic

load '../helpers/test_helper'

setup() {
    source "$(dirname "$BATS_TEST_FILENAME")/../helpers/test_helper.bash"
    export MAX_CALLS_PER_HOUR=100
    export CALL_COUNT_FILE=".call_count"
    export TEST_TEMP_DIR="$(mktemp -d /tmp/ralph-test.XXXXXX)"
    cd "$TEST_TEMP_DIR"
    echo "0" > "$CALL_COUNT_FILE"
}

teardown() {
    cd /
    rm -rf "$TEST_TEMP_DIR"
}

# Define the function being tested
can_make_call() {
    local calls_made=0
    [[ -f "$CALL_COUNT_FILE" ]] && calls_made=$(cat "$CALL_COUNT_FILE")
    [[ $calls_made -ge $MAX_CALLS_PER_HOUR ]] && return 1
    return 0
}

@test "can_make_call returns success when under limit" {
    echo "50" > "$CALL_COUNT_FILE"
    run can_make_call
    assert_success
}

Sources: TESTING.md:129-165

Test Helpers and Fixtures

The test suite is supported by a robust set of helper scripts located in tests/helpers/ that provide shared utilities, assertion functions, mock objects, and data fixtures.

Sources: TESTING.md:213-215

test_helper.bash

This file provides core assertion functions and setup utilities to standardize tests and reduce boilerplate.

Helper Function	Description
assert_success	Asserts that the last run command exited with status 0.
assert_failure	Asserts that the last run command exited with a non-zero status.
assert_equal "$actual" "$expected"	Compares two values for equality.
assert_file_exists "path/to/file"	Verifies that a file exists at the given path.
assert_valid_json "file.json"	Validates that a file contains well-formed JSON.
create_mock_prompt	Creates a sample PROMPT.md file for testing.
create_mock_fix_plan 5 2	Creates @fix_plan.md with 5 total tasks, 2 completed.

Sources: TESTING.md:219-245, CONTRIBUTING.md:191-201

fixtures.bash

This script is responsible for generating sample data and complete project structures for use in integration tests.

Fixture Function	Description
create_sample_prd_md "file.md"	Creates a sample Product Requirements Document in Markdown.
create_sample_fix_plan "@fix_plan.md" 10 3	Creates a sample fix plan with 10 tasks, 3 completed.
create_sample_status_running "status.json"	Creates a status.json file representing a running state.
create_sample_claude_output_success "output.log"	Creates a log file simulating a successful Claude Code execution.
create_test_project "project-name"	Creates a complete, mock Ralph project structure.

Sources: tests/helpers/fixtures.bash, TESTING.md:289-309

mocks.bash

To isolate tests from external dependencies, the suite uses a mocking system defined in mocks.bash. This allows for simulating the behavior of commands like the Claude Code CLI, git, and tmux. Mocks are controlled via environment variables.

The following diagram shows a test interacting with a mocked Claude Code CLI.

CI/CD Integration

The project uses GitHub Actions to automatically run the entire test suite on every push to main or develop, and on every pull request targeting main. The pipeline is defined in .github/workflows/test.yml.

This diagram shows the flow of the CI pipeline.

The pipeline consists of two main jobs:

1. test job: This is a required check. It installs dependencies and runs the unit and integration test suites. If any test fails, the build is marked as failed, and the pull request is blocked from merging.
2. coverage job: This job runs only after the test job succeeds. It executes the tests with kcov to generate a code coverage report. This report is uploaded as a build artifact for informational purposes but does not block the merge.

Sources: TESTING.md:349-361, CLAUDE.md:143-149

Conclusion

The testing strategy for Ralph is a cornerstone of its development process, ensuring that the autonomous agent remains stable, reliable, and correct in its behavior. By enforcing a 100% pass rate, organizing tests by scope, and providing a rich ecosystem of helpers and mocks, the project maintains a high standard of quality. The integration of this strategy into the CI/CD pipeline guarantees that no code is merged without passing a rigorous validation process, which is essential for a project designed to operate autonomously.

Sources: CONTRIBUTING.md:120-123, TESTING.md