Testing Strategy & Eval Harness
Relevant source files
The following files were used as context for generating this wiki page:
The testing strategy for RAGAPPv3 is designed to ensure the reliability of the Retrieval-Augmented Generation (RAG) pipeline, backend API integrity, and frontend component correctness. The project employs a multi-tiered approach encompassing backend unit and integration tests (using pytest), frontend unit and component tests (using Vitest and jsdom), and technical quality contracts. A critical component of this strategy is the Eval Harness, which specifically targets the E2E validation of the wiki and RAG integration, verifying citation accuracy and retrieval quality.
Sources: CONTRIBUTING.md:120-125, specs/wire-wiki-chat/SPEC.md:20-22
Testing Infrastructure and CI Gates
The project enforces strict quality gates through a CI pipeline defined in .github/workflows/ci.yml. These gates must be passed locally using the ci-compatibility-audit skill before any code is pushed or a PR is opened.
| Category | Tools & Frameworks | Key Requirements |
|---|---|---|
| Backend | Python 3.11, pytest, ruff | ruff check ., zero regression on targeted test subsets |
| Frontend | Node.js ≥ 20.19.0, Vitest, ESLint, TypeScript | npm run typecheck, npm run lint (max 0 warnings), npm test |
| Quality Contracts | Custom Python scripts | check_config_contract.py, check_pr_scope_drift.py |
Sources: CONTRIBUTING.md:79-110, AGENTS.md:43-46, CLAUDE.md:78-83
Frontend Testing Patterns
Frontend tests utilize Vitest with jsdom to simulate browser environments. Due to complexities in testing modern UI components, specific patterns are established for mocking and interaction.
Component Isolation and Mocking
Tests frequently mock the @/lib/api module to isolate components from network side effects. This includes mocking API base URLs and authentication helpers.
vi.mock("@/lib/api", () => ({
listWikiPages: vi.fn().mockResolvedValue({ pages: [], page: 1, per_page: 50 }),
API_BASE_URL: "/api",
getJwtAccessToken: vi.fn(() => null),
refreshAccessToken: vi.fn(),
}));
Sources: frontend/src/pages/WikiPage.test.tsx:10-30
Handling UI Library Limitations
Certain UI components, such as Radix Tabs or Select, cannot be triggered via standard fireEvent.click in jsdom. The strategy involves mocking these primitives with React Context to directly invoke onValueChange callbacks.
Eval Harness & E2E Validation
The evaluation strategy for the RAG system focuses on citation accuracy across three independent namespaces: Documents ([S#]), Memories ([M#]), and Wiki ([W#]).
Citation Validation Logic
The CitationValidator ensures that citations generated by the LLM are valid based on the retrieved evidence. This includes checking for "uncited factual warnings" and repairing content if citations are invalid.
Test Coverage Requirements
The "Wiki/Knowledge Compiler Chat Integration" feature requires extensive coverage to meet its quality gates:
- Backend Tests: Focus on FTS query normalization, query intent extraction, and
WikiRetrievalServicefallbacks. - Frontend Tests: 1092 tests total, including specific validations for
AssistantMessage(37 tests) andWikiCards(19 tests).
Sources: specs/wire-wiki-chat/SPEC.md:144-165
Server-Sent Events (SSE) Testing
Testing the chat stream and real-time updates requires simulating SSE connections. The project uses a controllableSse utility to push chunks to a stream reader, allowing for deterministic testing of job completion events.
Technical Quality Standards
- Python Version Policy: CI is pinned to Python 3.11. Tests failing on newer interpreters (e.g., 3.14) due to
RuntimeError: There is no current event loopare considered false failures. - Surgical Changes: Developers and agents must match existing styles and only modify code directly related to the task.
- Real Behavior Assertions: Tests must assert real behavior, including response bodies and state changes, rather than just HTTP status codes.
Sources: CLAUDE.md:85-92, CLAUDE.md:31-41, CONTRIBUTING.md:112-118
The testing strategy ensures that as new features like the Wiki Retrieval Service are added, they remain backward compatible with legacy sessions while providing high-fidelity, structured domain knowledge through a verified RAG pipeline.
Sources: specs/wire-wiki-chat/SPEC.md:20-28