AI Pipeline Smoke Testing
Relevant source files
The following files were used as context for generating this wiki page:
AI Pipeline Smoke Testing in the mdeapp project is a multi-layered verification strategy designed to ensure the integrity, accuracy, and performance of the AI-driven discovery and ticketing platform. It spans from low-level agent registration checks to high-level end-to-end (E2E) "synthetic" user flows that validate the integration between Mastra agents, Gemini LLM models, and external search APIs.
The testing suite focuses on several critical areas: agent configuration, query retrieval performance (Golden Queries), response faithfulness (hallucination detection), and production environment stability. These tests are executed via vitest for unit/smoke tests, tsx for specialized intelligence scripts, and playwright for E2E validation.
Sources: package.json:20-56, README.md:1-15, scripts/intelligence/faithfulness-smoke.ts:1-12
Testing Architecture and Tiers
The smoke testing framework is organized into distinct tiers, each targeting a specific layer of the AI pipeline.
Core Agent and Infrastructure Smoke Tests
At the lowest level, the system verifies that the Mastra agent runtime is correctly initialized and that all product agents (e.g., rentalAgent, eventAgent) are registered with the expected Gemini 3.5 Flash model. These tests also ensure that essential infrastructure, such as Supabase authentication files and environment variables, are present.
Sources: src/tests/smoke.test.ts:15-60
Intelligence and Quality Scoring
Specialized scripts evaluate the "intelligence" of the pipeline. This includes:
- Golden Queries: A set of predefined user queries (e.g., "quiet rooftop Provenza") tested against intelligent search libraries to verify ranking and hybrid search performance.
- Faithfulness Scorer: An LLM-free heuristic core that evaluates (toolOutputs, reply) pairs to detect hallucinations or fabricated information in assistant responses.
Sources: scripts/intelligence/faithfulness-smoke.ts:15-50, scripts/intelligence/golden-queries-smoke.ts:35-100
End-to-End (E2E) and Synthetic Smoke
The project utilizes Playwright to simulate real user interactions in both development and production environments. These tests monitor API responses from CopilotKit and Mastra to ensure that vertical-specific cards (rentals, events, restaurants) are rendered correctly and that the system does not suffer from "hijacking" where one search vertical incorrectly triggers another.
Sources: e2e/live-audit-verticals.spec.ts:38-120, e2e/prod-synthetic-smoke.spec.ts:25-70
Data Flow for Intelligence Verification
The following diagram illustrates the flow of a Golden Query through the intelligence testing scripts to determine if the pipeline meets accuracy and faithfulness thresholds.
This diagram shows how individual test cases are routed through specialized intelligent search functions and then validated against specific success criteria.
Sources: scripts/intelligence/golden-queries-smoke.ts:35-130, scripts/intelligence/faithfulness-smoke.ts:60-80
Key Test Suites and Commands
The project defines various smoke test categories within package.json.
| Command | Description | Relevant Files |
|---|---|---|
npm run smoke:golden-queries | Runs multi-vertical MIS Phase 1b checks for retrieval accuracy. | golden-queries-smoke.ts |
npm run smoke:faithfulness | Executes the hallucination/faithfulness heuristic scorer. | faithfulness-smoke.ts |
npm run test:e2e:live-audit | Playwright test for 4-vertical matrix (rentals, events, restaurants, cafes). | live-audit-verticals.spec.ts |
npm run test:e2e:prod-synthetic | Runs a 4-query matrix + POST budget check against production URL. | prod-synthetic-smoke.spec.ts |
npm run test | Standard vitest suite including core agent registration smoke tests. | src/__tests__/smoke.test.ts |
Sources: package.json:20-56
Faithfulness Evaluation Logic
The faithfulness scorer (evaluateFaithfulness) is a deterministic component that compares the final assistant reply against the actual tool outputs retrieved from the database or external APIs.
The faithfulness check is intentionally LLM-free to remain deterministic and safe to run in restricted environments without API keys.
Sources: scripts/intelligence/faithfulness-smoke.ts:10-14, 60-75
Production Synthetic Monitoring
The prod-synthetic-smoke.spec.ts suite performs a "matrix" test across four key verticals. It monitors the CopilotKit API to track the number of POST requests per query, which serves as a proxy for identifying inefficient loops or "run errors" in the production environment.
Performance Metrics Tracked
- Response Count: Number of
/api/copilotkitPOST requests per query. - UI Rendering: Verification of
[data-testid="rental-card"],event-card, andrestaurant-cardvisibility. - Idle Detection: Measurement of network activity after the Copilot agent becomes idle.
- Visual Evidence: Screenshots captured at each step and stored in
tmp/prod-synthetic-smoke.
Sources: e2e/prod-synthetic-smoke.spec.ts:33-85
Conclusion
AI Pipeline Smoke Testing ensures that the mdeapp remains "AI-first" by validating that its core agents are functional and its intelligence layer remains accurate. By combining heuristic faithfulness checks with E2E synthetic flows, the project maintains a high bar for retrieval quality and user experience across its key discovery verticals.
Sources: README.md:1-5, scripts/intelligence/golden-queries-smoke.ts:160-170