Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
Thoth employs a multi-tiered testing strategy to verify domain logic, infrastructure integration, and system lifecycle behavior. The project uses Vitest as the primary test runner across the monorepo, supplemented by specialized tools for architectural validation, security scanning, and specification compliance.
Tests are organized into unit, integration, and specialized "spec-check" categories. Integration tests frequently use a "stubbed service" approach to isolate interfaces (like MCP tools) while exercising the real transport layer.
Sources: apps/api/package.json:11-12, package.json:13-26, sdd-spec-check.config.toml:1-2
Test Execution Framework
The monorepo centralizes test execution through the root package.json, which delegates to individual workspaces (apps/api, apps/web, packages/utils).
| Command | Purpose | Tools Used |
|---|---|---|
pnpm test | Runs unit tests across all workspace packages. | Vitest |
pnpm test:integration | Runs integration tests specifically for API/Infrastructure. | Vitest (Integration project) |
pnpm spec-check | Validates that implementation matches documented SDD rules. | sdd-spec-check |
pnpm verify | Executes the full verification suite (Lint, Typecheck, Test, Build). | Turbo, Biome, Knip, Vitest |
Test Runner Configuration
The project configures Vitest with coverage support via @vitest/coverage-v8. The API application defines a specific integration project to separate long-running or infrastructure-heavy tests from unit tests.
Sources: package.json:13-26, apps/api/package.json:11-12, apps/api/package.json:54
Integration Testing Strategy
Integration tests in Thoth verify the interaction between HTTP routers, Model Context Protocol (MCP) tools, and the Dependency Injection (DI) container.
HTTP and MCP Integration
The HttpServer is tested using supertest to ensure routes like OpenAPI documentation (/openapi.json), Health checks, and MCP endpoints function correctly. MCP tests utilize a connectMcp helper to simulate client connections to tools registered within the server.
The diagram shows how integration tests exercise the HTTP layer and MCP routing while stubbing the underlying application services.
Sources: apps/api/test/infrastructure/http/server.integration.test.ts:44-120, apps/api/test/modules/bio/interface/bio.tool.integration.test.ts:32-40
Security and Middleware Testing
Middleware, specifically mcpAuthMiddleware, is tested to enforce access control rules:
- Rule MCP-007: Private tool calls without a bearer token return 401.
- Rule MCP-008: Private tool calls with a valid bearer token are permitted.
- Rule MCP-006: Private tool calls on handle-specific routes return 404 to prevent information leakage.
Sources: apps/api/test/infrastructure/http/mcp-auth.middleware.test.ts:59-130
Unit Testing Patterns
Domain Model Validation
Unit tests verify domain object constraints. For example, the Bio domain class is tested for its ability to reject blank fields and enforce valid property structures.
Sources: apps/api/test/modules/bio/domain/bio.test.ts:18-28
Application Service Logic
Services such as AgentService are tested with mocked repositories and external ports (e.g., LLM answering engines). Tests specifically check for:
- Standard successful flows (returning model answers).
- Error handling (e.g., returning 502 for empty LLM replies via
AGENT_EMPTY_REPLY). - Data persistence order (ensuring questions/replies are saved only after successful model response).
Sources: apps/api/test/modules/agent/application/agent.service.test.ts:79-165
Lifecycle and Reliability Testing
The LifecycleManager is tested to ensure reliable application startup and shutdown. This is critical for managing database connections and HTTP server state.
This diagram illustrates the error handling during the startup sequence as defined in RULE-LIFECYCLE-003.
Lifecycle Rules
| Rule ID | Behavior |
|---|---|
LIFECYCLE-001 | Services start in registration order. |
LIFECYCLE-002 | Services stop in reverse registration order. |
LIFECYCLE-004 | Each service stop() call is wrapped in a 10-second timeout. |
LIFECYCLE-007 | SIGTERM and SIGINT signals trigger a graceful stop. |
Sources: apps/api/test/lifecycle/manager.test.ts:50-130
Frontend Testing (Web)
The Web application uses Vitest and react-dom/server to verify server-side rendering (SSR) and API interaction logic.
- API Client Mocking:
getBiocalls are mocked to verify that the build fails if the backend returns invalid data or missing fields (Rules WEB-003, WEB-004, WEB-005). - Component Verification: Render tests ensure that bio information (name, headline, about) correctly appears in the HTML output.
- Revalidation Logic: Tests verify the ISR (Incremental Static Regeneration) timer is set correctly (e.g., 300 seconds as per
RULE-WEB-007).
Sources: apps/web/test/features/bio/bio.api.test.ts:28-54, apps/web/test/app/page.test.ts:14-25, apps/web/src/app/page.tsx:5
Architectural and Static Analysis
Beyond functional testing, the project enforces architectural and security standards through static analysis:
- Dependency Cruising:
depcruiseis used inapps/apiandapps/webto enforce module boundaries and prevent circular dependencies. - Secret Scanning:
secretlintruns against the entire monorepo to prevent the commitment of sensitive credentials. - Dead Code Detection:
knipidentifies unused files, exports, and dependencies. - Code Duplication:
jscpddetects clones to encourage code reuse across workspaces.
Sources: package.json:13-26, apps/api/package.json:14, apps/web/package.json:11
Summary
The Thoth testing strategy balances rapid feedback through unit tests with high-confidence integration tests using real HTTP transports and MCP protocols. By strictly linking test cases to "SDD Rules" (e.g., RULE-BIO-001), the project ensures that implementation remains synchronous with its technical documentation. The comprehensive pnpm verify pipeline guarantees that every contribution adheres to quality, security, and architectural standards before deployment.