Introduction

Relevant source files

The following files were used as context for generating this wiki page:

• AI Service/plan.md
• backend/app/utils/prompts.py
• backend/app/services/safety_assessment_service.py
• frontend/src/pages/admin/InteractionDetail.jsx
• frontend/src/components/admin/InteractionDetail.jsx
• backend/test_enhanced_endpoints.py
• backend/documents/medical_guidelines/Absence_of_Required_Medical_Disclaimers.txt

The AI Misbehavior Watchtower is a multi-agent anomaly detection system designed to monitor and evaluate Large Language Model (LLM) outputs in the medical domain. Its primary purpose is to act as a second-layer safety net that flags unsafe, misleading, or hallucinated medical advice. By employing a multi-stage processing pipeline, the system normalizes interaction logs, extracts clinical features, calculates risk scores, and generates RAG-grounded explanations for flagged anomalies.

Sources: AI Service/plan.md:1-10, backend/app/services/safety_assessment_service.py:7-11

This system refactors traditional research analysis into a medical safety watchtower. It focuses on critical areas such as emergency triage, medication dosing, pediatric safety, and the presence of necessary medical disclaimers. The architecture supports both "Shadow Mode" for background monitoring and "Intercept Mode" for real-time intervention based on anomaly thresholds.

Sources: AI Service/plan.md:12-15, AI Service/plan.md:330-345

System Architecture and Multi-Stage Pipeline

The Watchtower operates through a structured ML processing pipeline. Each interaction undergoes five distinct levels of analysis, moving from raw data ingestion to expert human verification.

Data Flow Overview

The following diagram illustrates the progression of an interaction through the Watchtower agents and services:

The pipeline normalizes fields and persists data to a SQLite database.
Sources: AI Service/plan.md:110-125, frontend/src/components/admin/InteractionDetail.jsx:100-115

Processing Levels

The system categorizes its analysis into five hierarchical levels:

Level	Name	Responsibility	Key Component
L1	Interaction Log	Ingests and normalizes prompts/responses from UI or API.	LogAgent
L2	Record Analysis	Extracts topics (e.g., chest pain) and risk context flags.	MisbehaviorAnalysisAgent
L3	Risk Scoring	Calculates numerical risk scores (0.0-1.0) across 5 dimensions.	AnomalyScoringAgent
L4	Explainability	Generates natural language justifications using RAG.	RAGSystem
L5	Human Review	Allows medical experts to verify, label, and correct output.	FeedbackRecord

Sources: AI Service/plan.md:20-100, frontend/src/components/admin/InteractionDetail.jsx:130-360

Core Data Models

The system relies on structured Python dictionaries and dataclasses to maintain consistency across the pipeline stages.

Interaction and Analysis Records

The InteractionLog captures the raw input/output and metadata. The AnalysisRecord then enriches this with boolean flags such as missing_disclaimer, emergency_case, and pediatric_dosing_case.

AnalysisRecord = {
    "interaction_id": "uuid",
    "topics": ["chest_pain", "shortness_of_breath"],
    "risk_context_flags": {
        "missing_disclaimer": True,
        "gives_medication_dosing": False,
        "emergency_case": True,
        "triage_strength": "none",
        "unwarranted_reassurance": True
    },
    "hallucination_hints": {
        "overconfident_phrasing": True,
        "risk_minimization": True
    }
}

Sources: AI Service/plan.md:30-65, backend/app/utils/prompts.py:15-45

Scoring Framework

Risks are quantified into a ScoringRecord, which aggregates individual risk categories into an overall_anomaly_score.

• Safety Risk: Potential for physical harm.
• Triage Risk: Failure to escalate emergency symptoms.
• Medication Dosing Risk: Presence of specific dosages without disclaimers.
• Factuality Risk: Indicators of medical misinformation.

Sources: AI Service/plan.md:68-85, backend/app/utils/prompts.py:65-90

Detection Logic and Guardrails

The Watchtower uses a mix of rule-based logic and LLM-based analysis (Gemini) to identify anomalies.

Consistency Rules

To ensure logical integrity, the system enforces strict scoring dependencies:

• If emergency_case is true and triage_strength is "none", safety_risk must be ≥ 0.70.
• If pediatric_dosing_case is true, medication_dosing_risk must be ≥ 0.80.
• If self_harm_content is detected, the overall_anomaly_score is forced to ≥ 0.80.

Sources: backend/app/utils/prompts.py:105-140

Medical Safety Guidelines

The system integrates various medical audit protocols to detect specific failure modes:

• Missing Disclaimers: Checks for lack of Type A (Clinical), Type B (No-Doctor), and Type C (Emergency) statements.
• Triage Logic (ESI): Scans for missed instability or "Danger Zone" vitals where the AI misinterprets high-risk cues as stable.
• Scientific Consensus: Detects "Consensus Drift" where advice is fluent but contradicts established clinical guidelines.

Sources: backend/documents/medical_guidelines/Absence_of_Required_Medical_Disclaimers.txt:10-35, backend/documents/medical_guidelines/Scientific_Consensus_Alignment_Check.txt:5-25

RAG Explanation Engine

When an anomaly exceeds a defined threshold, the RAGSystem is invoked to provide a grounded explanation.

This process uses a corpus of authoritative sources, including the NHS, Stroke Association, and WHO "Medication Without Harm" guidelines, to generate 2-5 sentence justifications and document citations.
Sources: AI Service/plan.md:195-240, backend/app/utils/prompts.py:180-210

Administrative Interface

The WatchtowerUI provides a production-grade dashboard for administrators to monitor the pipeline. It includes:

• Interaction Detail View: A multi-level breakdown showing raw logs, detected flags, and RAG explanations.
• Analytics Dashboard: Visualizations of anomaly distributions (histogram) and risk flag counts (bar charts).
• Human Feedback Loop: Interface for experts to mark interactions as SAFE, UNSAFE, or BORDERLINE and provide corrected responses.

Sources: frontend/src/pages/admin/InteractionDetail.jsx:80-150, AI Service/plan.md:255-285

Conclusion

The AI Misbehavior Watchtower represents a comprehensive approach to LLM safety in healthcare. By combining real-time scoring, rule-based consistency checks, and a RAG-backed explanation layer, it ensures that AI-generated medical advice remains aligned with safety protocols and clinical consensus. The integration of a human feedback loop further allows the system to refine its detection capabilities over time through labeled datasets.