Watchtower UI
Relevant source files
The following files were used as context for generating this wiki page:
The Watchtower UI is a multi-layered interface system designed for monitoring, analyzing, and auditing AI medical outputs for safety anomalies. It encompasses both a terminal-based management interface (WatchtowerUI) for orchestrating backend analysis tasks and a sophisticated React-based administrative dashboard for granular inspection of flagged interactions.
The primary purpose of the Watchtower UI is to surface misbehaviors in LLM medical advice, such as missing disclaimers, unsafe medication dosing, or failed emergency triaging. It acts as the visualization layer for a multi-agent detector that generates RAG (Retrieval-Augmented Generation) grounded explanations for every detected risk.
Sources: AI Service/plan.md:1-12, AI Service/watchtower/ui.py:15-25
System Architecture and Components
The Watchtower UI follows a modular architecture that separates data ingestion, automated analysis, and human-in-the-loop verification. It integrates directly with the LogAgent for data handling and the WatchtowerOrchestrator for executing detection pipelines.
The diagram above illustrates the relationship between the UI components and the underlying agentic services.
Sources: AI Service/plan.md:158-166, AI Service/watchtower/ui.py:20-25
Core Components
| Component | Description | Source File |
|---|---|---|
WatchtowerUI | Python class managing the CLI menu, log ingestion, and analytics plotting. | AI Service/watchtower/ui.py:16 |
InteractionDetail | React component providing a 5-level deep dive into specific AI interactions. | frontend/src/components/admin/InteractionDetail.jsx:68 |
PipelineStage | A UI wrapper used to visualize the discrete steps of the ML processing pipeline. | frontend/src/components/admin/InteractionDetail.jsx:32 |
DimensionScores | Visual representation of risk across quality, hallucination, alignment, and safety. | frontend/src/pages/admin/InteractionDetail.jsx:132 |
Administrative Dashboard Features
The web-based administrative UI provides a highly detailed "Interaction Detail" view, which organizes data into a sequential pipeline. This view allows auditors to trace an interaction from the raw input to the final human review.
Multi-Level Inspection Pipeline
The UI structure reflects the system's processing stages:
- Interaction Log: Displays the original prompt, AI response, and metadata (timestamp, model, user ID).
- Record Analysis: Shows extracted medical topics and safety flags (e.g.,
emergency_case,self_harm_content). - Risk Scoring: Visualizes automated scores across multiple dimensions like
safety_riskandfactuality_risk. - Explainability Layer: Presents RAG-generated explanations and the specific citations used to justify a flag.
- Human Review: Captures expert verification, corrected responses, and reviewer comments.
Sources: frontend/src/components/admin/InteractionDetail.jsx:116-290, frontend/src/pages/admin/InteractionDetail.jsx:105-320
The sequence shows the data flow from an admin selecting an interaction to submitting human-in-the-loop feedback.
Sources: frontend/src/pages/admin/InteractionDetail.jsx:16-55, AI Service/watchtower/ui.py:173-195
Management Interface (CLI)
The CLI component of Watchtower UI provides administrative controls for managing the lifecycle of anomaly detection.
Menu Operations
The management interface supports the following operations:
- Ingest Log: Manually input prompts and responses for testing or backfilling.
- Run Analysis: Execute the orchestrator over a batch of logs using
shadow(monitoring only) orintercept(blocking) modes. - View Anomalies: A filtered view showing interactions that exceed the defined
anomaly_threshold. - Analytics Dashboard: Generates visual plots using
matplotlibto track score distributions and flag counts. - Feedback Labeling: Allows auditors to mark interactions as
SAFE,UNSAFE, orBORDERLINE.
Sources: AI Service/watchtower/ui.py:30-45, AI Service/plan.md:200-210
Anomaly Visualization Logic
The UI includes automated plotting capabilities to monitor system health:
- Score Distribution: Histogram of
overall_anomaly_score. - Flag Counts: Bar chart of the most frequent safety violations.
- Time-Series: Line chart tracking anomaly frequency over time.
Sources: AI Service/watchtower/ui.py:200-260
Data Models and Schemas
The UI relies on several standardized data structures to ensure consistency across the Python backend and React frontend.
Scoring Record
{
"interaction_id": "uuid",
"scores": {
"safety_risk": 0.0,
"factuality_risk": 0.0,
"triage_risk": 0.0,
"medication_dosing_risk": 0.0,
"overall_anomaly_score": 0.0
},
"flags": {
"missing_disclaimer": true,
"emergency_case": false,
"overconfident_diagnosis": true
}
}
Sources: AI Service/plan.md:46-65, backend/app/utils/prompts.py:146-168
Feedback Schema
| Field | Type | Description |
|---|---|---|
interaction_id | UUID | Link to the original interaction. |
human_label | String | One of SAFE, UNSAFE, or BORDERLINE. |
corrected_response | String | Optional expert-written response for future training. |
comments | String | Notes from the medical reviewer. |
Sources: AI Service/plan.md:80-88, AI Service/watchtower/ui.py:184-192
Technical Implementation Details
Detection Layer UI (Enhanced)
The React frontend includes specialized logic for "Enhanced Detection" which breaks down anomalies into five specific dimensions:
- Quality Anomaly: Issues with response structure or coherence.
- Hallucination Anomaly: Claims that lack verification against ground truth.
- Alignment Anomaly: Mismatches between the response and the medical context.
- Safety Anomaly: Direct clinical risks or missing critical triage.
- Confidence Anomaly: Poorly calibrated model confidence (e.g., being overconfident on uncertain topics).
Sources: frontend/src/pages/admin/InteractionDetail.jsx:144-230, frontend/src/components/admin/InteractionDetail.jsx:218-235
Source Citations in UI
For transparency, the UI renders RAG citations derived from the ExplanationRecord. Each citation includes:
doc_id: The source document (e.g., "NHS_EMERGENCY_Triage").chunk_id: The specific text segment.score: The relevance of that snippet to the flagged anomaly.
Sources: AI Service/plan.md:68-76, frontend/src/components/admin/InteractionDetail.jsx:274-282
Summary
The Watchtower UI provides a comprehensive toolkit for managing AI safety in medical contexts. By combining a high-level CLI for orchestration and an in-depth web dashboard for forensic analysis, the system enables technical users and medical experts to effectively monitor, audit, and improve LLM outputs. The architecture emphasizes explainability through RAG citations and ensures a robust human-in-the-loop feedback mechanism to refine detection performance over time.