Testing Strategy & Fixtures
Relevant source files
The following files were used as context for generating this wiki page:
Vantage employs a multi-layered testing strategy to verify the reliability of its data collection pipeline, AI drafting engine, and administrative controls. The suite uses vitest to execute unit and integration tests, relying heavily on environment variable manipulation and service mocking to simulate various provider states.
Sources: package.json:20, test/social-provider-routing.test.ts:43-48
Provider Escalation and Mocking
The system tests a prioritized hierarchy of data providers. Tests verify that collectors attempt free or low-cost retrieval methods before escalating to paid agents or APIs. Fixtures serve as the final fallback for demonstration and development modes.
This diagram illustrates the prioritized escalation logic verified in provider routing tests.
Sources: test/social-provider-routing.test.ts:79-130, test/tinyfish-collectors.test.ts:88-125
Provider Coverage States
Tests classify collector results into specific health and coverage categories to ensure the UI reflects data provenance accurately.
| Coverage Type | Condition | Health Status |
|---|---|---|
healthy | Live data retrieved via standard provider | healthy |
degraded | Results returned from static fixtures or partial scans | degraded |
access_pending | Missing API keys (e.g., SCAVIO_API_KEY) | degraded |
budget_limited | Rate limits or budget exhaustion reached | degraded |
Sources: test/source-coverage.test.ts:13-64, test/collector-runner.test.ts:104-113
AI Drafting and Quality Validation
Drafting tests verify the "grounding" of AI-generated content. The validation logic ensures every factual claim in a draft corresponds to a specific quote in the provided evidence.
Grounding Verification Flow
The system rejects drafts that invent information or reintroduce prohibited promotional language.
Testing confirms that each sentence in the draft matches a provided evidence claim ledger.
Sources: test/contribution-quality.test.ts:40-62, lib/drafting/contribution.ts:42-80
AI Triage and Cost Management
Tests verify that the system uses cheaper models for triage before escalating to premium models for final drafting. The suite simulates cached token scenarios to verify cost calculation accuracy.
Sources: test/contribution-model-routing.test.ts:32-50, lib/drafting/contribution.ts:111-122
Browser Simulation and Security
Vantage uses a specialized fake-browser utility to test browser-based collection without spawning real headless instances. Tests focus on security boundaries and cost recording.
- SSRF Guards: The suite verifies that the
browserRunmodule rejects internal or local IP addresses before invoking the Cloudflare Browser binding. - Platform Denylist: Tests ensure the system refuses to scrape prohibited social platforms (e.g., LinkedIn, Facebook) to comply with community rules.
- Cost Accounting: Tests confirm that wall-clock time or
X-Browser-Ms-Usedheaders correctly calculate USD costs for billed operations.
Sources: test/browser-run.test.ts:51-118, test/browser-run.test.ts:175-188
Collector State and Persistence
Tests for the collector-runner ensure that source state (etags, cursors) persists correctly across runs, even when a scan fails or returns zero documents.
The testing suite verifies atomic deduplication and state preservation logic.
Sources: test/collector-runner.test.ts:133-162, test/source-coverage.test.ts:71-78
Administrative Control Testing
The system tests administrative routes to ensure cross-origin protection and strict authorization.
- Origin Checking: Tests reject requests with invalid
Originorsec-fetch-siteheaders. - Audit Logging: Tests verify that every administrative switch flip (e.g., pausing Reddit collection) creates a corresponding audit log entry in the same database batch.
Sources: test/source-switch-admin.test.ts:56-83, test/source-switch-admin.test.ts:117-128
Summary
Vantage's testing strategy prioritizes grounding for AI drafts, escalation for data collection, and privacy for internal error handling. By using a combination of static fixtures and dynamic mocks, the suite ensures that the system behaves predictably under various API failure modes and budget constraints.