Intelligence Evaluation & Scorers

Intelligence Evaluation & Scorers represent the framework within the mdeapp project dedicated to measuring the quality, accuracy, and grounding of AI agent responses. The primary objective is to detect hallucinations—instances where the AI invents entities, prices, or IDs not present in the source data—and to ensure that all mentions of venues or listings are supported by tool outputs.

The system utilizes a hybrid approach combining deterministic heuristic checks with optional LLM-based "judges" (specifically using Gemini 3.5 Flash). These scorers are registered via the Mastra framework, allowing them to be monitored through dedicated API endpoints and used in CI/CD pipelines to maintain a high "faithfulness rate" across agent interactions.

Sources: src/mastra/scorers/faithfulness.ts:18-35, src/app/api/scorers/route.ts:5-11

Scorer Architecture

The evaluation system is built on the Mastra createScorer utility, which structures the evaluation process into discrete steps: preprocessing, analysis, score generation, and reasoning. This modularity allows for "fail-closed" logic where the system can fall back to pure heuristics if an LLM judge is unavailable or fails.

Core Scoring Pipeline

The following diagram illustrates the data flow within a typical scorer, such as the Faithfulness Scorer.

The pipeline ensures that claims are first extracted and compared against tool data using deterministic code before calling an LLM to demote potential false positives.
Sources: src/mastra/scorers/faithfulness.ts:18-31, src/mastra/scorers/grounding-coverage.ts:72-88

Registered Scorers

The project currently implements two primary scorers registered in the central scorers/index.ts registry.

Sources: src/mastra/scorers/index.ts:7-11, src/mastra/scorers/faithfulness.ts:98-102, src/mastra/scorers/grounding-coverage.ts:66-70

Faithfulness Scorer (AGT-00A)

This scorer focuses on strict data integrity. It identifies "claims" within an agent's response—specifically entities, prices, and IDs—and verifies their existence in the toolOutputs provided to the agent. It uses Gemini 3.5 Flash as a judge to verify if a claim is a genuine fabrication or merely a paraphrase/synonym.
Sources: src/mastra/scorers/faithfulness.ts:18-31, src/mastra/scorers/faithfulness.ts:59-71

Grounding Coverage Scorer (AGT-00B)

This scorer measures the precision of entity mentions. It filters extracted claims to focus only on "entity" types (e.g., venue names). The score is calculated as the ratio of grounded entity mentions to total entity mentions. A score of 1.0 indicates perfect grounding.
Sources: src/mastra/scorers/grounding-coverage.ts:11-15, src/mastra/scorers/grounding-coverage.ts:89-94

Data Models and Storage

The evaluation results and scorer definitions are persisted in a Supabase database. The schema includes tables for storing individual scorer runs and the versions of agent configurations.

Database Schema for Evaluation

The database tracks every evaluation run, including the prompts used for extraction and reasoning, allowing for historical auditing of agent performance.
Sources: src/lib/supabase/database.types.ts:1688-1758, src/lib/supabase/database.types.ts:1620-1686

API Endpoints

The project exposes a public GET route to list available scorers, primarily for use by operations and CI.

GET /api/scorers

Returns a list of all registered AI-quality scorers on the Mastra instance.

Response Body Structure:

{
  "count": 2,
  "scorers": [
    {
      "key": "faithfulness",
      "id": "faithfulness",
      "name": "Hallucination / Faithfulness",
      "description": "..."
    }
  ]
}

Sources: src/app/api/scorers/route.ts:16-25

Smoke Testing and CI

The scripts/intelligence/faithfulness-smoke.ts script provides a "golden set" of fixtures to validate the scorers. It includes test cases for:

• Grounded replies: Expected score 1.0.
• Fabricated listings: Expected to be flagged (score < 1.0).
• Invented prices: Expected to be flagged.
• Clarifying questions: Expected score 1.0 (no checkable claims).

The smoke tests are designed to be deterministic and can run offline without a Gemini API key by relying on the LLM-free heuristic core.
Sources: scripts/intelligence/faithfulness-smoke.ts:14-60

Conclusion

The Intelligence Evaluation & Scorers system provides a critical safety layer for the mdeapp AI agents. By combining deterministic claim extraction with LLM-backed verification, the system ensures that discovery and ticketing information provided to users for Medellín remains grounded in reality, effectively mitigating the risks of AI hallucination in financial (prices) and logistical (venue names) contexts.