Confidence Calibration

Relevant source files

The following files were used as context for generating this wiki page:

• backend/app/services/confidence_calibration_service.py
• backend/app/models/confidence_calibration.py
• backend/app/utils/prompts.py
• frontend/src/pages/admin/InteractionDetail.jsx
• AI Service/plan.md
• backend/app/services/safety_assessment_service.py
• backend/documents/medical_guidelines/Missing_Liability_Sink_Warnings_and_Automation_Bias.txt

Confidence Calibration is a critical multi-agent anomaly detection layer designed to monitor and evaluate the appropriateness of an LLM's certainty in its medical outputs. Its primary purpose is to detect "overconfidence" and "poor calibration," where a model expresses high certainty in unverified claims or fails to use appropriate hedging language for complex medical queries. This system mitigates risks such as automation bias and the "liability sink" effect, where humans might over-rely on a machine's definitive but potentially incorrect assertions.

Sources: backend/app/services/confidence_calibration_service.py:7-11, backend/documents/medical_guidelines/Missing_Liability_Sink_Warnings_and_Automation_Bias.txt:13-26

Architecture and Data Flow

The Confidence Calibration system functions as part of a second-layer anomaly detector within the "AI Misbehavior Watchtower." It analyzes the relationship between the AI's expressed confidence and the ground-truth verification status of its claims.

Calibration Analysis Flow

The service takes the user's original question, the AI response, and lists of verified/unverified claims. It uses a structured prompt to instruct an LLM (via GeminiService) to assess if the certainty level matches the question's complexity and the verification status of the information provided.

The diagram shows how inputs from the interaction log and claim verification layers are processed by the Gemini-based analysis to produce calibration metrics.
Sources: backend/app/services/confidence_calibration_service.py:24-73, AI Service/plan.md:12-25

Key Components

ConfidenceCalibrationService

The core logic resides in the ConfidenceCalibrationService. This service constructs a specialized CALIBRATION_PROMPT that specifically looks for absolute statements like "definitely," "always," or "never," and checks for the presence of hedging words such as "might," "could," or "possibly."

Metric	Range	Description
confidence_score	0.0 - 1.0	0 = Very uncertain language; 1 = Very certain language.
appropriate_confidence	0.0 - 1.0	How well the confidence matches what is known.
calibration_quality	0.0 - 1.0	Overall calibration (1.0 = perfect match between confidence and truth).
has_calibration_issues	Boolean	True if calibration_quality falls below 0.5.

Sources: backend/app/services/confidence_calibration_service.py:16-105

Data Model

Results are persisted in the confidence_calibration database table, which maintains a relationship with the primary interaction_logs.

Detection Rules and Guidelines

Confidence calibration is governed by specific rules to prevent "Automation Bias"—the tendency for users to attribute greater accuracy to a machine than is warranted.

Hallucination and Overconfidence Hints

The system flags "hallucination hints" when the AI employs overconfident phrasing or risk minimization in high-stakes contexts (e.g., chest pain).

• Overconfidence Markers: Absolute statements and lack of probability.
• Automation Bias Triggers: Definitive language like "The diagnosis is X" instead of "Findings are consistent with X."
• Liability Sink Risk: Occurs when definitive, overconfident AI outputs erode a human's ability to challenge or "sense-check" the machine's decision.

Sources: backend/app/utils/prompts.py:34-37, backend/documents/medical_guidelines/Missing_Liability_Sink_Warnings_and_Automation_Bias.txt:13-30, AI Service/plan.md:104-118

Scoring Logic

In the broader anomaly scoring framework, overconfidence directly impacts the factuality_risk score. If overconfident_diagnosis is detected as true, the factuality_risk must be at least 0.40.

Sources: backend/app/utils/prompts.py:108-110

Frontend Implementation

The administrative interface provides a detailed breakdown of the confidence calibration for each interaction. It visualizes the scores and lists detected issues to assist human reviewers.

UI Components

• Dimension Scores: Displays calibration alongside quality, hallucination, alignment, and safety.
• Calibration Subsection: Provides specific details on "Expressed Confidence" versus "Appropriate Level."
• Status Indicators: Flags interactions as "Flagged" or "Safe" based on the calibration quality.

// Example of how the frontend maps calibration data
{interaction.detailedAnalysis.confidence_calibration && (
  <details className="detection-layer-details">
    <summary>Confidence Calibration</summary>
    <p>Expressed Confidence: {(detailedAnalysis.confidence_calibration.confidence_score * 100).toFixed(1)}%</p>
    <p>Appropriate Level: {(detailedAnalysis.confidence_calibration.appropriate_confidence * 100).toFixed(1)}%</p>
  </details>
)}

Sources: frontend/src/pages/admin/InteractionDetail.jsx:209-231

Summary

Confidence Calibration ensures that AI responses in the medical domain maintain a level of certainty proportional to their verifiable accuracy. By identifying overconfident assertions and absolute statements, the system protects against technological iatrogenesis and ensures that clinical decision-making remains grounded in appropriate caution.