HUD Persona Fidelity Evals

Relevant source files

The following files were used as context for generating this wiki page:

• src/evals/personaFidelity.ts
• src/test-smoke.ts
• src/test-demo-site.ts
• plan.md
• src/engine/promptCompiler.ts
• README.md

HUD Persona Fidelity Evals is a benchmarking system designed to verify that synthetic browser agents behave consistently with their assigned PersonaConfig traits. It validates that an agent's internal monologue, navigation choices, and reactions during a browsing session align with the statistical profile generated for that specific user persona.

This module provides behavioral expectation checks against actual agent events, generating a score based on passed checks. These results are pushed to the HUD SDK to provide developers with a clear view of how realistically the AI agents are simulating the target Ideal Customer Profile (ICP).

Sources: plan.md:93-97, README.md:16-25, src/evals/personaFidelity.ts:1-10

Architecture and Data Flow

The fidelity evaluation occurs after an agent has completed its navigation task. It processes a collection of AgentStepEvent objects—which contain the agent's evaluation, memory, and next goals—and compares them against the original PersonaConfig.

Evaluation Logic Flow

The following diagram illustrates how raw agent events are transformed into a fidelity score and reported to the HUD infrastructure.

The engine aggregates text from evaluation, memory, and next_goal fields to search for trait-specific keywords.
Sources: src/evals/personaFidelity.ts:19-25, src/evals/personaFidelity.ts:51-140

Key Data Structures

The system relies on several core interfaces to define the evaluation output.

Interface	Field	Type	Description
FidelityCheck	name	string	The ID of the specific trait being validated.
	passed	boolean	Whether the agent's behavior matched the trait.
	reason	string	Technical justification for the result.
FidelityResult	score	number	Ratio of passed checks to applicable checks.
	checks	FidelityCheck[]	Array of individual trait validations.
	applicable_checks	number	Number of traits in the config that had valid checks.

Sources: src/evals/personaFidelity.ts:4-17

Behavioral Validation Rules

The fidelity engine uses a set of heuristic checks mapped to the 19 traits defined in the persona model. These checks look for specific signals in the agent's telemetry.

Logic Mappings

Trait	Condition	Success Criteria
tech_literacy	< 0.3	Mentions UI confusion (e.g., "lost", "stuck", "what does").
tech_literacy	> 0.8	Absence of basic UI confusion keywords.
price_sensitivity	> 0.7	Visits a URL containing "pricing" or mentions "cost", "budget", "$".
patience	< 0.3	Total step count is below the population average.
trust_baseline	< 0.3	Mentions trust/security keywords (e.g., "secure", "legitimate", "scam").
goal_clarity	< 0.3	Visits more than 2 distinct URL paths (exploration).

Sources: src/evals/personaFidelity.ts:55-140

Keyword Categories

The evaluator maintains internal constant arrays to identify behavioral signals:

• UI_CONFUSION_KEYWORDS: confus, unclear, hesitat, overwhelm, etc.
• TRUST_KEYWORDS: secure, privacy, testimonial, badge, etc.
• JARGON_KEYWORDS: acronym, technical, complex language, etc.

Sources: src/evals/personaFidelity.ts:27-50

Integration with HUD SDK

The results are exported to the HUD platform via a subprocess bridge. This allows the fidelity scores to be integrated into the broader project tracing and monitoring infrastructure.

The pushToHUD function uses child_process.execSync to pipe a JSON payload into a Python script that interacts with the hud library.
Sources: src/evals/personaFidelity.ts:153-186

Usage in Pipeline

Fidelity evaluations are integrated into the main execution loops, such as test-demo-site.ts and test-smoke.ts.

// Example from src/test-demo-site.ts
for (const r of results) {
  const fidelity = evaluatePersonaFidelity(
    compiled[r.persona_index]!.config, 
    r.steps, 
    avgSteps
  );
  console.log(`Agent ${r.persona_index} fidelity: ${fidelity.score.toFixed(2)}`);
}

Sources: src/test-demo-site.ts:100-105, src/test-smoke.ts:81-86

Summary

HUD Persona Fidelity Evals serve as the "ground truth" validator for the SynthCrowd platform. By checking if an impatient, low-tech persona actually expresses frustration or struggles with navigation, the system ensures that the synthetic testing results are statistically significant and realistic reflections of real human behavior. These metrics allow developers to trust the friction reports and code change recommendations generated by the platform.
Sources: plan.md:15-20, README.md:120-125