Hallucination Detection
Relevant source files
The following files were used as context for generating this wiki page:
- backend/app/services/hallucination_detection_service.py
- backend/app/models/hallucination_detection.py
- backend/documents/medical_guidelines/Detection_of_Hallucinated_Medical_Citations_and_Data.txt
- backend/app/utils/prompts.py
- AI Service/plan.md
- backend/documents/medical_guidelines/Scientific_Consensus_Alignment_Check.txt
- frontend/src/pages/admin/InteractionDetail.jsx
The Hallucination Detection system is a multi-stage audit framework designed to identify "medically plausible but factually incorrect information" generated by AI models. In the context of this project, medical hallucinations include the fabrication of scientific literature, citations, clinical findings, or anatomical data that contradict established ground truth or scientific consensus. Sources: backend/documents/medical_guidelines/Detection_of_Hallucinated_Medical_Citations_and_Data.txt:1-12, backend/app/services/hallucination_detection_service.py:1-12
The system utilizes a Retrieval-Augmented Generation (RAG) approach to cross-reference AI-generated claims against a trusted knowledge base of medical guidelines and scientific indices. It serves as a second-layer "Watchtower" that monitors LLM outputs, flags anomalies, and provides grounded explanations for why a specific response is considered unreliable. Sources: AI Service/plan.md:1-8, backend/app/services/hallucination_detection_service.py:33-40
Detection Architecture
The detection process is managed by the HallucinationDetectionService, which orchestrates three primary stages: claim extraction, RAG-based verification, and final risk assessment.
Processing Pipeline
- Stage 1: Factual Claim Extraction: The service uses an LLM (Gemini) to isolate verifiable factual claims from the response, categorizing them by type (e.g., medical fact, statistical, recommendation) and confidence level.
- Stage 2: RAG Verification: Extracted claims are queried against the guideline knowledge base. Claims with a similarity score above a threshold (typically 0.7) are marked as verified.
- Stage 3: Risk Assessment: The system analyzes unverified claims, looking for markers of fabrication such as overconfident phrasing, specific numbers provided without sources, or hedging words.
Sources: backend/app/services/hallucination_detection_service.py:42-100, backend/app/utils/prompts.py:175-200
System Data Flow
The following diagram illustrates the lifecycle of a hallucination check from initial AI output to stored analysis results.
The diagram shows the sequential logic of extracting, verifying, and scoring medical claims. Sources: backend/app/services/hallucination_detection_service.py:58-95
Hallucination Taxonomy
Anomalies are classified into specific types to clarify the nature of the error for auditors and human reviewers.
| Type | Name | Mechanism / Risk |
|---|---|---|
| Type A | Citation Fabrication | Generating references that look valid (journal name, DOI) but do not exist. |
| Type B | Multimodal/Imaging | False identification of anatomical structures or pathological findings (e.g., claiming a fracture exists in a normal scan). |
| Type C | Consensus Failure | Providing linguistically coherent advice that contradicts established medical practice guidelines. |
| Type D | Intention Mismatch | Failing to understand the "underlying intention" of a query, leading to "technically fluent" but harmful advice. |
Sources: backend/documents/medical_guidelines/Detection_of_Hallucinated_Medical_Citations_and_Data.txt:14-43, backend/documents/medical_guidelines/Scientific_Consensus_Alignment_Check.txt:41-55
Data Models
The system persists detection results in the HallucinationDetection model, which links to the primary interaction logs.
HallucinationDetection Fields
| Field | Type | Description |
|---|---|---|
id | String (FK) | Unique identifier linked to interaction_logs.id. |
extracted_claims | JSON | List of claims with text, confidence, and type. |
verified_claims | JSON | Claims successfully matched against RAG sources. |
unverified_claims | JSON | Claims that failed verification, including RAG scores. |
hallucination_risk_score | Float | Final risk score from 0.0 (Safe) to 1.0 (High Risk). |
hallucination_markers | JSON | Specific warning signs (e.g., "overconfident_phrasing"). |
recommended_action | String | Directive for the system: flag, review, or safe. |
Sources: backend/app/models/hallucination_detection.py:15-38, backend/app/services/hallucination_detection_service.py:84-100
Audit Metrics and Scoring
The system uses specific metrics to quantify hallucination severity, notably the ACHMI (Assessment of Caption Hallucinations in Medical Imagery) for descriptive responses.
Scoring Logic
- ACHMI_I (Instance Level): Calculated as
|Hallucinated Components| / |All Medical Components|. A high score indicates a high density of false entities. - ACHMI_S (Sentence Level): Calculated as
|Captions with Hallucinations| / |All Captions|. A high score indicates frequent unreliable statements. - General Hallucination Score: Derived from the
AnomalyScoringAgent, which applies weights to factuality risk (0.1 weight) and overconfident diagnosis (which triggers a factuality risk boost).
Sources: backend/documents/medical_guidelines/Detection_of_Hallucinated_Medical_Citations_and_Data.txt:85-102, AI Service/plan.md:144-165, backend/app/utils/prompts.py:85-110
Mitigation and Human Review
When the detection risk score exceeds specific thresholds, the system triggers mitigation strategies to prevent patient harm.
Recommended Actions
- Constraint to Evidence: Forcing the generator to cite only from the RAG context window, prohibiting the use of parametric knowledge.
- Qualified Human Intervention: Flagging for review by a licensed physician before the response is finalized.
- Disclaimer Injection: Appending warnings about AI limitations and the potential for fabrication.
Sources: backend/documents/medical_guidelines/Detection_of_Hallucinated_Medical_Citations_and_Data.txt:111-120, backend/documents/medical_guidelines/Scientific_Consensus_Alignment_Check.txt:69-79
Admin Interface Visualization
The administrative frontend provides a "Multi-Dimensional Detection" view where auditors can expand layers to see claims analysis, including the number of verified vs. unverified claims and specific hallucination markers.
The diagram represents the relationship between the main interaction view and the detailed hallucination analysis component in the UI. Sources: frontend/src/pages/admin/InteractionDetail.jsx:136-168
Summary
Hallucination Detection in this architecture is not a single check but an integrated pipeline. By extracting factual assertions and verifying them against high-authority guidelines, the system moves beyond simple pattern matching to a semantic verification of AI accuracy. This serves as a critical safety buffer in medical applications where fabrication can lead to direct clinical harm. Sources: backend/app/services/hallucination_detection_service.py:14-30, AI Service/plan.md:10-15