Interaction Review Interface
Relevant source files
The following files were used as context for generating this wiki page:
The Interaction Review Interface is a critical component of the AI Misbehavior Watchtower, designed to facilitate human-in-the-loop oversight of medical AI interactions. Its primary purpose is to allow expert reviewers to verify automated anomaly detections, provide safety labels, and suggest corrected responses for interactions flagged by the multi-agent detection system. Sources: AI Service/plan.md:1-10, frontend/src/pages/AdminReviewPage.jsx:8-15
The system follows a multi-level architectural approach where automated analysis (including safety scoring and RAG-based explanations) is presented to the reviewer alongside the original interaction log to enable informed decision-making. Sources: frontend/src/pages/admin/InteractionDetail.jsx:84-110, frontend/src/components/admin/InteractionDetail.jsx:105-120
System Architecture and Data Flow
The review workflow begins when the automated pipeline identifies a high-risk interaction and persists it in the scoring_records and analysis_records tables. The human review data is subsequently captured in the feedback table. Sources: AI Service/plan.md:76-85, AI Service/plan.md:144-148
Review Data Lifecycle
This diagram illustrates the progression of an interaction from automated flagging to human feedback submission.
Sources: AI Service/plan.md:214-225, frontend/src/components/admin/ReviewInterface.jsx:25-35
Core Components
1. Review Interface Form
The ReviewInterface component provides the interactive form used by administrators to label interactions. It supports three primary labels: SAFE, UNSAFE, and BORDERLINE. Sources: frontend/src/components/admin/ReviewInterface.jsx:6-7
| Feature | Description | File Reference |
|---|---|---|
| Human Label | Toggle buttons to select the safety classification (SAFE, UNSAFE, BORDERLINE). | ReviewInterface.jsx:75-90 |
| Corrected Response | Optional text area for providing an ideal response if the AI output was unsafe. | ReviewInterface.jsx:92-101 |
| Reviewer Comments | Optional field for additional context or rationale behind the review. | ReviewInterface.jsx:103-112 |
| Locking Mechanism | Disables inputs once submitted unless the "Edit Review" mode is activated. | ReviewInterface.jsx:46-60 |
2. Interaction Detail Pipeline
The InteractionDetail component displays the multi-stage analysis generated by the ML pipeline. It is organized into levels:
- Level 1 (Log): The raw user prompt and AI response.
- Level 2 (Analysis): Extracted medical topics and safety flags (e.g.,
emergency_case,missing_disclaimer). - Level 3 (Scoring): Numerical risk scores for toxicity, medical accuracy, and safety.
- Level 4 (Explanation): RAG-based justification for why the item was flagged.
- Level 5 (Review): Displays existing human feedback if available.
Sources: frontend/src/components/admin/InteractionDetail.jsx:105-280, frontend/src/pages/admin/InteractionDetail.jsx:84-250
3. Flagged Items List
A specialized dashboard view that aggregates high-risk interactions. It uses an "Anomaly Score" (0-100) to prioritize items for the reviewer. Sources: frontend/src/components/admin/FlaggedItemsList.jsx:38-55
Review Logic and State Management
The interface manages several states to ensure data integrity during the review process:
Sequence of actions during a feedback submission event.
Sources: frontend/src/components/admin/ReviewInterface.jsx:25-40, frontend/src/pages/AdminReviewPage.jsx:36-47
Data Structures
The review feedback is structured as a FeedbackRecord object:
FeedbackRecord = {
"interaction_id": "uuid",
"human_label": "SAFE|UNSAFE|BORDERLINE",
"corrected_response": "optional str",
"comments": "optional str",
"timestamp": "ISO-8601",
"reviewer_id": "optional"
}
Sources: AI Service/plan.md:65-72
Anomaly Detection Integration
The review interface directly reflects the outputs of the Gemini-based agents. For instance, if the LEVEL_3_SCORING_PROMPT results in an overall_anomaly_score above a specific threshold, the interface highlights the item in the "Flagged Review" queue. Sources: backend/app/utils/prompts.py:60-75, frontend/src/components/admin/FlaggedItemsList.jsx:75-85
The overall_anomaly_score is calculated as a weighted sum:
- Safety Risk: 35%
- Triage Risk: 25%
- Medication Dosing Risk: 20%
- Factuality Risk: 10%
- Anxiety Inducing Risk: 10%
Sources: AI Service/plan.md:175-185
Conclusion
The Interaction Review Interface serves as the final arbiter in the medical safety watchtower. By bridging automated multi-agent analysis with human expertise, it ensures that high-risk AI outputs are identified, corrected, and used as training data for future model alignment. Its modular design allows for detailed inspection of every stage of the ML processing pipeline before a final human label is applied. Sources: AI Service/plan.md:214-220, frontend/src/components/admin/InteractionDetail.jsx:100-110