Evaluations & Analytics
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Evaluations & Analytics system in tldw_server is a unified engine designed to measure and monitor the quality of AI-generated content, including summaries, RAG (Retrieval-Augmented Generation) responses, and general LLM outputs. It provides a standardized framework for grading AI performance through metrics such as fluency, consistency, relevance, and faithfulness. This system allows developers and end-users to perform batch scoring, run comparative benchmarks, and track quality trends over time to ensure production-ready outputs.
The module is integrated deeply with the core LLM orchestration and storage layers, utilizing its own dedicated database for logs and metrics. It supports multiple evaluation methodologies, including G-Eval for summaries and specialized RAG metrics for question-answering systems. By providing both REST API endpoints and a WebUI interface, the system facilitates a "grade-and-improve" workflow essential for refining research assistants and media analysis pipelines.
Sources: Docs/Architecture.md, AGENTS.md, Docs/User_Guides/Server/Evaluations_End_User_Guide.md
System Architecture & Data Flow
The Evaluations system operates as a core module within the FastAPI backend, interacting with external LLM providers to perform "LLM-as-a-judge" grading. It maintains a centralized SQLite database (evaluations.db) to store metrics and audit logs, separate from the per-user content databases.
High-Level Component Interaction
The following diagram illustrates how the evaluation service interacts with other system components during a grading request.
The system distinguishes between ad-hoc evaluations (direct grading of provided text) and benchmark runs (grading against a pre-defined dataset).
Sources: Docs/Architecture.md, AGENTS.md, CLAUDE.md
Evaluation Methodologies
The system supports three primary categories of evaluation, each tailored to specific content types and quality requirements.
1. G-Eval (Summary Quality)
G-Eval is used primarily for grading summaries. It evaluates content across four distinct dimensions, typically using a 1-5 scale.
| Metric | Range | Description |
|---|---|---|
| Fluency | 1-3 | Evaluates the quality of writing and grammar. |
| Consistency | 1-5 | Measures factual alignment with the original source text. |
| Relevance | 1-5 | Assesses whether the summary includes all important points. |
| Coherence | 1-5 | Checks for logical flow and organizational structure. |
Sources: Docs/User_Guides/Server/Evaluations_End_User_Guide.md
2. RAG Evaluation
Specifically designed for Retrieval-Augmented Generation, this evaluation checks the relationship between a user query, the retrieved source context, and the final generated response.
- Faithfulness: Verifies if the answer is derived solely from the retrieved context without hallucinations.
- Answer Relevance: Checks if the response directly addresses the user's question.
- Answer Similarity: Compares the generated output against a "gold standard" or ideal answer if provided.
Sources: Docs/User_Guides/Server/Evaluations_End_User_Guide.md
3. Benchmarking System
Benchmarks allow for standardized testing of specific models or prompts against a static catalog of data. The benchmark workflow follows a specific sequence in the API and WebUI.
API Implementation & Configuration
The Evaluations API is mounted under the /api/v1/evals (or /api/v1/evaluations) namespace. It handles both individual item processing and batch operations.
Key API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/evaluations/geval | POST | Performs G-Eval scoring on a source/summary pair. |
/evaluations/rag | POST | Evaluates a RAG response based on query and context. |
/evaluations/batch | POST | Processes multiple items in parallel to optimize cost and latency. |
/evaluations/benchmarks | GET | Lists all available pre-defined benchmarks. |
/evaluations/benchmarks/{name}/run | POST | Triggers a benchmark run with specific model/limit params. |
Sources: Docs/User_Guides/Server/Evaluations_End_User_Guide.md, Docs/Plans/2026-03-02-benchmark-guide-api-webui-extension-design.md
Example Request Structure
Evaluations typically require the source text, the generated output, and the provider configuration.
{
"name": "test_evaluation",
"eval_type": "exact_match",
"eval_spec": { "threshold": 1.0 },
"dataset": [
{
"input": { "output": "test_result" },
"expected": { "output": "test_result" }
}
]
}
Sources: Helper_Scripts/generate_docs_with_config.py
Database & Persistence
Evaluation data is stored in Databases/evaluations.db. This database uses a centralized schema (not per-user) to facilitate cross-system analytics and auditing.
- Schema Focus: The DB tracks evaluation IDs, timestamps, metric scores, and the raw reasoning provided by the LLM judges.
- Performance: For high-volume environments, the system implements rate limiting for LLM calls (utilizing
slowapior module-specific limiters) to prevent provider exhaustion during batch runs.
Sources: Docs/Architecture.md, AGENTS.md, CLAUDE.md
Conclusion
The Evaluations & Analytics module provides a critical feedback loop for the tldw_server ecosystem. By standardizing the measurement of AI outputs through G-Eval, RAG metrics, and automated benchmarks, the system ensures that generated research and media summaries remain accurate and relevant. Its architecture balances immediate ad-hoc grading with scalable batch processing, backed by a persistent analytics store that enables long-term quality assurance and model comparison.