Chat API & Orchestrators
Relevant source files
The following files were used as context for generating this wiki page:
The Chat API and Orchestrator system serves as the core intelligence layer of the AI Misbehavior Watchtower. Its primary purpose is to provide a multi-layered, agentic framework for monitoring Large Language Model (LLM) medical outputs. By intercepting or shadowing chat interactions, the orchestrator detects unsafe advice, hallucinations, and triage failures through a series of specialized analysis and scoring agents.
The system is designed as a "second-layer" anomaly detector. It processes raw interaction logs—comprising user prompts and AI responses—and applies rule-based logic combined with LLM-driven critiques to generate safety flags, risk scores, and RAG-grounded explanations. This ensures that medical symptom-checking remains within safe boundaries and complies with clinical guidelines.
Sources: AI Service/plan.md:1-10, AI Service/watchtower/ui.py:92-120
Pipeline Architecture
The system operates as a multi-stage processing pipeline where data flows through specialized agents to reach a final safety determination.
This diagram illustrates the flow from initial log ingestion to the automated analysis pipeline and the eventual human-in-the-loop feedback mechanism.
Sources: AI Service/plan.md:120-135, frontend/src/pages/admin/InteractionDetail.jsx:136-160
Core Components
1. LogAgent & Data Schema
The LogAgent is responsible for normalizing and persisting interaction data. It handles fields such as prompt, response, model name, and metadata (tags like "medical" or "symptoms").
| Data Record | Purpose | Key Fields |
|---|---|---|
| InteractionLog | Primary record of chat | prompt, response, model_name, metadata |
| AnalysisRecord | Feature extraction results | topics, risk_context_flags, hallucination_hints |
| ScoringRecord | Quantitative risk values | safety_risk, triage_risk, overall_anomaly_score |
| ExplanationRecord | Root cause analysis | risk_type, explanation, citations |
Sources: AI Service/plan.md:15-60, AI Service/watchtower/ui.py:53-70
2. Misbehavior Analysis Agent
This agent identifies medical topics and flags specific risk contexts using predefined rules and LLM prompts. It categorizes risks such as emergency cases, pediatric dosing, or missing disclaimers.
Detection Categories:
- Emergency Symptoms: Patterns like chest pain or stroke signs.
- Specialized Populations: Cues for pregnancy, pediatrics, or elderly users.
- Medication Patterns: Regex-based dosage detection and drug class keywords.
Sources: AI Service/plan.md:143-175, backend/app/utils/prompts.py:8-35
3. Anomaly Scoring Agent
The scoring agent converts analysis flags into quantitative risks (0.0 to 1.0). It follows strict consistency rules to ensure that high-risk flags (e.g., self_harm_content) result in high overall_anomaly_score values (≥ 0.80).
# Weighted score composition example
overall = weighted_sum([
safety_risk * 0.35,
triage_risk * 0.25,
medication_dosing_risk * 0.2,
factuality_risk * 0.1,
anxiety_inducing_risk * 0.1
])
Sources: AI Service/plan.md:195-215, backend/app/utils/prompts.py:40-120
API & Orchestration Modes
The orchestrator supports different operational modes to balance safety with user experience.
Operational Modes
- Shadow Mode: The system monitors and logs anomalies in the background without affecting the live user response.
- Intercept Mode: If the
overall_anomaly_scoreexceeds a defined threshold, the orchestrator can block the response or inject a warning.
Admin API Endpoints
The backend provides several endpoints for retrieving the results of the orchestration pipeline:
GET /api/admin/metrics/enhanced: Global safety metrics.GET /api/admin/analytics/anomaly-breakdown: Breakdown of anomalies over time.GET /api/admin/interactions/{id}/detailed: Full multi-stage analysis for a specific interaction.
Sources: backend/test_enhanced_endpoints.py:48-115, AI Service/plan.md:320-330
Retrieval-Augmented Generation (RAG) Layer
When an interaction is flagged as an anomaly, the RAGSystem generates a human-readable explanation by querying a corpus of medical safety guidelines (e.g., NHS, WHO, BNFC).
This sequence ensures that every safety flag is backed by authoritative medical documentation, reducing the "black box" nature of AI safety decisions.
Sources: AI Service/plan.md:230-275, backend/app/utils/prompts.py:125-155
Conclusion
The Chat API and Orchestrator system provides a robust framework for medical LLM safety. By decoupling the core interaction from the safety analysis, the system allows for complex, multi-agent evaluations including rule-based checks, quantitative scoring, and RAG-based explainability. This architecture is critical for maintaining clinical safety standards and providing transparency in AI-driven healthcare assistance.
Sources: AI Service/plan.md:340-350, frontend/src/pages/admin/InteractionDetail.jsx:136-150