Test Scripts & Validation Suite
Relevant source files
The following files were used as context for generating this wiki page:
The Test Scripts & Validation Suite provides a comprehensive set of tools to verify the SynthCrowd persona engine, browser agent execution, and the end-to-end optimization pipeline. These scripts range from isolated unit-style checks for persona generation to full smoke tests that integrate multiple third-party services like Anthropic, Convex, MongoDB, and Browser Use.
The suite ensures that the statistical variance in customer populations remains realistic, that compiled system prompts produce actionable behavioral rules, and that parallel browser agents correctly report friction points and conversion outcomes. Sources: src/test-smoke.ts:1-10, src/test-validation.ts:1-10, README.md
Testing Architecture & Logic
The testing suite is organized into different levels of integration, from pure logic validation to full-stack execution.
Persona Engine Validation
These tests focus on the transition from Natural Language (NL) inputs to structured PersonaConfig objects and their subsequent compilation into system prompts.
The logic verifies that different NL inputs (e.g., "Grandma in Ohio" vs "SF Tech Bro") produce meaningfully different configurations in fields like tech_literacy, patience, and decision_style. Sources: src/test-validation.ts:13-35, src/test-population.ts:31-45
Full Pipeline Integration
The full pipeline tests execute the entire flow, including agent navigation and report generation. This involves:
- Interpretation: Converting ICP descriptions to a center-point persona.
- Population: Generating N variants using Gaussian distribution.
- Parallel Execution: Running agents in batches with staggered delays.
- Reporting: Synthesizing agent events into friction points and recommendations.
Sources: src/test-full-pipeline.ts:3-25, src/test-parallel.ts:14-30
Key Test Scripts
The following table summarizes the primary test scripts available in the repository:
| Script | Purpose | Key Dependencies |
|---|---|---|
test-validation.ts | Validates interpreter variety, prompt contrast, and population spread. | Anthropic |
test-smoke.ts | Verifies integration with all 7 core tools (Laminar, Convex, MongoDB, etc.). | Full Stack |
test-pipeline.ts | Tests the basic NL to compiled agent flow without browser execution. | Anthropic |
test-parallel.ts | Executes 10 agents in parallel batches to test concurrency and streaming. | Browser Use, Convex |
test-demo-site.ts | Runs a read-only analysis against a demo website, proposing but not applying code changes. | Anthropic, FS |
Sources: package.json:6-14, src/test-smoke.ts:147-156, src/test-demo-site.ts:109-125
Smoke Test Integration Flow
The smoke test (test-smoke.ts) provides a sequential verification of the entire ecosystem.
This sequence ensures that every service in the stack is reachable and functional before deploying to production or running large-scale tests. Sources: src/test-smoke.ts:25-130
Validation Metrics
The suite uses specific statistical and behavioral checks to ensure quality:
Population Spread
Tests verify that the generatePopulation function maintains a proper distribution. For a population of 8 with a variance of 0.6, the suite calculates the mean and standard deviation for traits like age and tech_literacy to ensure outliers are created (noted as [O] or [OUTLIER] in logs). Sources: src/test-validation.ts:103-115, src/test-population.ts:50-60
Persona Fidelity (HUD)
The system checks if the agent's actions match its assigned persona. This is performed by evaluatePersonaFidelity, which compares the agent's behavioral rules against actual step events.
- Passed Checks: Rules the agent successfully followed.
- Applicable Checks: Total rules relevant to the observed behavior.
- Fidelity Score: Calculated as
passed_checks / applicable_checks.
Sources: src/test-smoke.ts:88-95, src/test-demo-site.ts:79-84
Execution Environment Configuration
Tests require specific environment variables to function correctly. If these are missing, scripts are designed to skip the relevant steps gracefully.
| Variable | Required For |
|---|---|
ANTHROPIC_API_KEY | ICP Interpretation, Report Generation, Code Analysis |
BROWSER_USE_API_KEY | Browser Agent Navigation |
CONVEX_URL | Real-time event streaming and result storage |
MONGODB_URI | Long-term persona and run persistence |
LMNR_PROJECT_API_KEY | Laminar tracing of LLM calls |
Sources: src/test-smoke.ts:16-115, README.md
Summary
The Test Scripts & Validation Suite acts as the quality assurance layer for SynthCrowd. By combining statistical validation of personas with real-world browser execution and multi-service smoke tests, it ensures the reliability of the synthetic user testing and automated code optimization loops.