Basin Analyzer MCP
Relevant source files
The following files were used as context for generating this wiki page:
The Basin Analyzer MCP is a specialized Model Context Protocol server designed for context drift detection and AI output confidence measurement. It addresses the challenge of LLM hallucination and instability by analyzing how response patterns cluster into "basins of attraction." By running queries multiple times and measuring the semantic distance between outputs, the server provides a quantitative reliability score for AI-generated content.
In the broader architecture of the eng-platform, the Basin Analyzer serves as a quality control layer within the research pipeline. It sits between the orchestration layer (e.g., research-swarm) and the final output generation (e.g., report-writer), ensuring that only stable and coherent information is promoted to finalized reports.
Sources: mcp-servers/README.md:21-65, mcp-servers/README.md:104-106
Core Concepts: Basin Analysis
Basin analysis treats LLM outputs as a landscape where stable response patterns represent "basins." If an LLM consistently returns similar semantic meanings across multiple samples, it has found a deep "basin," indicating high confidence. Conversely, high variance in outputs indicates a lack of stability, often caused by overloaded context windows or ambiguous prompts.
Key Metrics
The analyzer calculates several critical metrics to determine output stability:
| Metric | Range | Description |
|---|---|---|
| Epsilon (ε) | 0 - 1 | Measures the distance between basins; lower values indicate higher confidence. |
| Basin Count (n) | 1+ | The number of distinct semantic clusters found in the response set. |
| Coherence | 0 - 1 | Overall consistency of the responses; higher values indicate better stability. |
| Confidence Level | Category | A classification (HIGH, MEDIUM, or LOW) based on combined metrics. |
Sources: mcp-servers/README.md:112-117, CHANGELOG.md:223-233
Query Classification
The server categorizes queries based on their expected basin behavior:
- FACTUAL: Single correct answer; expected low epsilon and 1 basin.
- INTERPRETIVE: Multiple valid views; expected medium epsilon and 2-3 basins.
- COUNTERFACTUAL: Hypothetical scenarios; expected high epsilon and 3+ basins.
Sources: mcp-servers/README.md:119-122, CHANGELOG.md:235-238
System Architecture
The Basin Analyzer follows the Model Context Protocol (MCP) and integrates into a research swarm for automated validation.
This diagram illustrates the flow from a research request through the internal clustering logic of the Basin Analyzer before reaching the report generation phase.
Sources: mcp-servers/README.md:21-65, mcp-servers/README.md:144-148
Implementation Details
The server is implemented in Python and utilizes semantic embeddings and LLM calls to perform its analysis. It requires external API keys for embedding generation and model inference.
Configuration and Setup
The Basin Analyzer requires specific environment variables for semantic processing:
VOYAGE_API_KEY: Used for semantic embeddings to measure similarity between response clusters.DEEPSEEK_API_KEY: Used for the underlying LLM calls during the sampling process.
The server typically runs on port 9383.
Sources: mcp-servers/README.md:104, mcp-servers/README.md:135-141, PROMPT-ADD-TO-EXISTING.md:109-111
MCP Tools
The server exposes several tools to MCP clients for measuring and classifying AI stability:
| Tool | Purpose |
|---|---|
measure_confidence | Runs a query N times to measure semantic stability and clustering. |
analyze_query_type | Classifies a query as Factual, Interpretive, or Counterfactual. |
get_basin_metrics | Retrieves detailed data regarding cluster distances and counts. |
compare_responses | Performs semantic similarity checks between two specific outputs. |
Sources: mcp-servers/README.md:144-148
Performance and Use Cases
Performance Benchmarks
- Analysis Time: Approximately 30 seconds for a sample size of N=5.
- Accuracy: 92% correlation with human-rated confidence levels.
- Cost Efficiency: Utilizes Voyage API for embeddings ($0.001/query) and DeepSeek-Chat for inference ($0.01/query).
Sources: mcp-servers/README.md:250-254
Primary Use Cases
- Report Validation: Automatically checking AI-generated reports for hallucinations before they are published.
- Context Monitoring: Detecting when an LLM's context window is overloaded, which typically manifests as high output drift.
- Prompt Engineering A/B Testing: Measuring which prompt variations result in the most stable "basins" for a given task.
- RAG Pipeline Quality: Monitoring the consistency of Retrieval-Augmented Generation outputs over time.
Sources: mcp-servers/README.md:129-133
Conclusion
The Basin Analyzer MCP provides a mathematically grounded approach to AI reliability within the eng-platform. By quantifying response stability through epsilon metrics and basin clustering, it enables developers to build more deterministic and trustworthy AI-native workflows.
Sources: mcp-servers/README.md:112-122, README.md:203-207