Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
Roughly implements a multi-layered testing and verification strategy designed for an AI-agent-driven development environment. Since the project itself is composed of Markdown-based skill and agent definitions rather than traditional compiled code, the testing strategy focuses on structural invariants, project-specific verification loops, and human-in-the-loop validation gates.
The strategy is bifurcated into internal project verification (testing Roughly itself) and consumer project verification (how Roughly tests the code it builds). Internal verification relies on structural drift checks and dogfooding scenarios, while consumer verification utilizes the verify-all skill and automated subagent review loops.
Sources: CLAUDE.md:33-36, README.md:162-167
Consumer Project Verification
Roughly's primary mechanism for ensuring code quality in user projects is a tiered verification pipeline that runs during implementation (/roughly:build or /roughly:fix) and as a standalone check.
The verify-all Loop
The verify-all skill is a standardized verification loop that executes project-specific commands defined in a project's CLAUDE.md. It provides a three-step check sequence that iterates until successful or a fix cap is reached.
| Check | Command Source | Purpose |
|---|---|---|
| Type Check | {{TYPE_CHECK_COMMAND}} | Static type verification (e.g., tsc, mypy) |
| Test Suite | {{TEST_COMMAND}} | Unit and integration tests (e.g., npm test, pytest) |
| Build | {{BUILD_COMMAND}} | Production build verification (e.g., npm run build, go build) |
Sources: skills/verify-all/SKILL.md:25-45, skills/setup/SKILL.md:95-103
Automated Fix Attempts
When a verification check fails, the system attempts to auto-fix the issue before escalating to the human operator.
- Implementer Auto-fix: During task implementation, the orchestrator allows up to 4 attempts for type-check/lint failures and 2 attempts for test failures.
- Verify-all Auto-fix: The standalone loop allows up to 3 fix attempts per check.
Sources: skills/fix/SKILL.md:179-195, skills/verify-all/SKILL.md:52-57
Internal Structural Verification
For its own codebase, Roughly uses a Stop hook (.claude/hooks/verify-all.sh) to enforce structural invariants every time the LLM completes a turn.
The diagram shows the internal structural checks performed by the project's Stop hook to maintain plugin integrity.
Sources: CONTRIBUTING.md:209-234
Structural Invariants
Roughly enforces specific constraints to ensure agents and skills remain within context window limits and maintain consistent behavior:
- Skill Line Limit: Skills must stay under 300 lines to prevent orchestrator confusion.
- Agent Word Limit: Agent prompts are capped at 650 words to maintain focus.
- Pre-flight Identity: Hard-abort skills must contain byte-identical migration check blocks to ensure safety across the pipeline.
Sources: CLAUDE.md:46-47, CONTRIBUTING.md:218-225
Dogfooding and CI
Roughly is tested using a "dogfood" strategy where the plugin is used to build its own features. The Continuous Integration (CI) pipeline automates this process.
CI Scenarios
The CI pipeline (scripts/ci-dogfood.sh) executes specific scenarios to verify plugin stability:
- Smoke Test: Verifies CLI plumbing and basic connectivity.
- Auth Failure: Ensures the system handles invalid API keys gracefully.
- Happy-path Build: Runs a full
/roughly:build --cicycle in a minimal target project (tests/fixtures/hello-roughly/).
Sources: CONTRIBUTING.md:172-200
The --ci Flag
The --ci flag enables non-interactive testing. When active, it alters the standard 8-stage pipeline behavior:
- Stage 4 (Review Plan): Skips the interactive review loop. In
build, it emits a syntheticPASS. Infix, it acts on thereview-planverdict without a human gate. - Stage 8 (Wrap-up): Auto-progresses commits without requiring manual approval.
Sources: skills/fix/SKILL.md:43-51, CONTRIBUTING.md:188-193
Plan and Epic Validation
Validation occurs before any code is written through specialized review agents and subagents.
Review-Plan Subagent
Before implementation, every plan is verified by the review-plan subagent. It checks for:
- Completeness: Requirement gaps and task verification steps.
- Assumptions: Validating file paths and API existence.
- Overengineering: Identifying unnecessary abstractions.
Sources: skills/review-plan/SKILL.md:28-60
Epic Review and Audit
Epics undergo validation at both ends of the lifecycle:
- Pre-implementation (
review-epic): Uses theepic-reviewer(Opus) to check for cross-story dependencies and technical feasibility. - Post-implementation (
audit-epic): Verifies that every Acceptance Criterion (AC) in the epic isMETby mapping stories to files via git history and auditing the results.
Sources: skills/review-epic/SKILL.md:26-38, skills/audit-epic/SKILL.md:41-80
Summary
Roughly's testing strategy replaces traditional unit tests with multi-stage verification gates and structural invariant enforcement. By combining automated subagent reviews (review-plan, review), project-specific verification loops (verify-all), and internal drift checks (verify-all.sh), the project ensures both its own structural integrity and the implementation quality of the features it generates.