Testing Strategy & Fixtures

Relevant source files

The following files were used as context for generating this wiki page:

• design/testing.md
• .claude/rules/testing.md
• src/test/pglite.ts
• src/llm/anthropic.test.ts
• src/mcp/adapter.test.ts
• CLAUDE.md
• CONTRIBUTING.md

The Cogmo project employs a multi-tiered testing strategy designed to ensure reliability across deterministic logic, database interactions, and non-deterministic LLM integrations. The strategy emphasizes local development speed using in-process tools like PGlite alongside robust integration and end-to-end (E2E) suites powered by Testcontainers and record/replay mocks.

The testing architecture is built on Vitest for high-speed TypeScript execution and utilizes specialized tools like llmock to simulate Anthropic and OpenAI-compatible APIs. This ensures that the agentic loop can be validated without incurring actual API costs or dealing with the latency of live LLM providers.
Sources: design/testing.md:1-10, CLAUDE.md:330-335

Three-Tier Testing Architecture

Cogmo categorizes tests into three distinct layers, each serving a specific purpose in the development lifecycle.

1. Unit Tests

Unit tests focus on deterministic logic and isolated components. They use Vitest and in-process mocks to maintain high execution speed.

• Store Implementations: Validated against real SQL using PGlite (PostgreSQL 17 WASM) to ensure Drizzle queries are correct.
• LLM Calls: Uses Zod schema validation and SDK mocking to test retry logic and response parsing.
• Adapter Modules: Mock the transport layer to test setup and inbound message handling.

Sources: design/testing.md:12-25, .claude/rules/testing.md:1-10

2. Integration Tests

Integration tests exercise the orchestration pipeline by wiring application modules in-process while using Docker-based infrastructure.

• Infrastructure: Testcontainers manage PostgreSQL, Redis, Inngest, and Hindsight with random ports.
• LLM Simulation: llmock runs in-process to serve Anthropic and OpenAI-compatible endpoints.
• Scope: Tests the full pipeline from event arrival (e.g., inbound/arrived) to assistant message persistence in the database.

Sources: design/testing.md:29-45, .claude/rules/testing.md:12-18

3. E2E Tests

E2E tests serve as smoke tests for the full deployment stack, running Cogmo as a subprocess in "connect mode."

• Migrations: Verifies that the application correctly applies database migrations on boot.
• Functional Check: Emits a real event via the Inngest API and asserts that the expected response appears in the database.

Sources: design/testing.md:47-58

Testing Tier Summary

Tier	File Suffix	Tools Used	Primary Goal
Unit	.test.ts	Vitest, PGlite, llmock (local)	Logic isolation & speed
Integration	.integration.test.ts	Testcontainers, llmock	Pipeline orchestration
E2E	.e2e.test.ts	Subprocess spawn, Inngest dev server	Full stack smoke test

Sources: design/testing.md:12-58, CONTRIBUTING.md:65-72

Database Testing with PGlite

Cogmo uses PGlite, an in-process PostgreSQL 17 WASM implementation, to test store implementations without requiring a full Docker container for every unit test.

The diagram shows the lifecycle of a database-backed unit test using the PGlite utility.
Sources: src/test/pglite.ts:1-20, design/testing.md:27-28

Key functions in the PGlite utility include:

• createTestDatabase(): Initializes a fresh PGlite instance, ensures pg_uuidv7 is available, and pushes the current Drizzle schema.
• truncateAll(db): Clears all tables to provide a clean slate for the next test case, ensuring test isolation.

Sources: src/test/pglite.ts:25-45, design/testing.md:27-28

Mocking and Fixtures

For external dependencies that are non-deterministic or network-dependent, Cogmo relies on a strict record/replay and mocking strategy.

LLM Mocking (llmock)

The project utilizes @copilotkit/llmock to handle LLM requests. This allows developers to:

1. Record: Run tests against live APIs to capture real JSON responses.
2. Replay: Use recorded fixtures in CI to ensure deterministic builds and zero API costs.
3. Mock: Manually define responses for specific scenarios like tool use or thinking blocks.

Sources: design/testing.md:88-95, .claude/rules/testing.md:20-25

Recorded Fixture Example

Recorded fixtures contain the exact match criteria (model, user message, turn index) and the corresponding model response, including tool calls and metadata.
Sources: test/fixtures/recorded/anthropic-2026-05-19T18-16-01-158Z-3118469e.json:1-20

External Service Strategy

Service	Strategy	Implementation
Anthropic API	Fixture-based mock	llmock server with streaming/tool_use support
Ollama/OpenAI	Fixture-based mock	llmock serving OpenAI-compatible endpoints
MCP Servers	Client Stubbing	Stub MCP client returning fixed tool results
Telegram	Framework Mocking	vi.mock("grammy") in unit tests

Sources: design/testing.md:88-95, src/mcp/adapter.test.ts:40-55

Coverage Patterns & Guards

Cogmo enforces specific test recipes to handle common failure modes in agentic systems.

JSONB and Schema Robustness

Since Cogmo uses PostgreSQL JSONB columns extensively, tests must verify both write-side Zod enforcement and read-side resilience against malformed raw SQL updates.
Sources: design/testing.md:63-70

Non-Settling Promise Guards

To prevent CI flakes and "unbounded" hangs, Cogmo uses a pattern of structurally bounding promises to ensure they reject within a specific deadline.

// Correct pattern for timeout testing
const err = await handle.wait().catch((e: Error) => e);
expect(err).toBeInstanceOf(ExecTimeoutError);

Sources: design/testing.md:112-126

Concurrency Invariants

Tests are required for in-process pub/sub systems to pin semantics for:

• Subscribers added mid-emit.
• Self-unsubscription during an emission.
• Burst ordering across multiple subscribers.

Sources: design/testing.md:130-136

Local Workflow

Developers are expected to run the following sequence to validate changes:

1. pnpm test: Runs unit tests (PGlite, mocked LLM).
2. pnpm test:record: Re-records LLM fixtures if prompts or tools changed.
3. pnpm test:integration: Runs orchestration tests against Testcontainers.

Sources: CONTRIBUTING.md:14-25, CLAUDE.md:330-335

The project’s CI environment validates these tiers on every PR, blocking merges if LLM requests do not match recorded fixtures or if coverage gaps are detected in new behavioral branches.
Sources: CONTRIBUTING.md:65-75, CLAUDE.md:265-275

The Cogmo testing strategy ensures that while the agent's behavior may evolve, the underlying infrastructure and orchestration remain stable through rigorous verification of both deterministic code and non-deterministic model interactions.