LLM Integration (Gemini & Ollama)

Relevant source files

The following files were used as context for generating this wiki page:

• backend/app/services/gemini_service.py
• backend/app/utils/prompts.py
• backend/app/services/explanation_agent.py
• backend/app/services/context_alignment_service.py
• backend/app/services/safety_assessment_service.py
• AI Service/plan.md

Introduction

The LLM integration system acts as the core intelligence layer of the AI Anomaly Detection Watchtower. It leverages Large Language Models—primarily Google Gemini—to perform multi-stage analysis, risk scoring, and RAG-based (Retrieval-Augmented Generation) explanations for medical chatbot interactions. The system is designed to identify "misbehavior" such as missing medical disclaimers, unsafe medication dosing, and failure to escalate emergency symptoms.

This module refactors traditional ingestion systems into a multi-agent orchestrator. These agents process raw interaction logs and transform them into structured AnalysisRecords, ScoringRecords, and ExplanationRecords. While Gemini is the primary driver for high-stakes clinical auditing, the architecture supports multi-model interactions including local deployments for processing or future expansions.

Sources: AI Service/plan.md:3-8, backend/app/services/gemini_service.py:12-20

Multi-Agent Pipeline Architecture

The integration follows a structured 5-level processing pipeline where LLMs are called at specific stages to perform technical and clinical audits.

Data Flow Diagram

The following diagram illustrates how an interaction log moves through the LLM-powered agents to generate a final safety assessment.

The diagram shows the sequence from raw data ingestion to the final human-in-the-loop review stage.
Sources: AI Service/plan.md:143-155, backend/app/services/explanation_agent.py:32-45

Gemini Service Implementation

The GeminiService class provides the interface for both standard text generation and strict structured data extraction.

Core Functions

Function	Description
generate_response	Generates conversational text with medical context system prompts.
generate_json	Enforces strict JSON output, utilizing regex fallbacks if the model returns markdown fences.
stream_response	Supports token-by-token streaming for future WebSocket implementations.

Sources: backend/app/services/gemini_service.py:27-113

Model Configuration

The service primarily utilizes the gemini-2.5-flash-lite model for rapid analysis and JSON generation. It includes error handling that catches "model not found" errors and falls back to stable versions if specific experimental models fail.
Sources: backend/app/services/gemini_service.py:16-19, backend/app/services/gemini_service.py:141-145

Specialized Analysis Agents

LLM prompts are categorized into specific "Levels" to ensure modularity and accurate scoring.

Level 2: Analysis & Feature Extraction

The LLM acts as a medical safety analyst, identifying clinical topics and hallucination hints.

• Prompting: Uses LEVEL_2_ANALYSIS_PROMPT to extract binary flags for cases like emergency_case or gives_medication_dosing.
• Logic: Errs on the side of safety; if any risk is plausible, flags are set to true.

Sources: backend/app/utils/prompts.py:7-40

Level 3: Risk Scoring & Consistency

This stage calculates numeric scores (0.0 to 1.0) across five dimensions.

• Dimensions: Safety Risk, Factuality Risk, Triage Risk, Medication Dosing Risk, and Anxiety Inducing Risk.
• Consistency Rules: The LLM is instructed with strict conditional logic (e.g., if pediatric_dosing_case is TRUE, then medication_dosing_risk MUST be ≥ 0.60).

Sources: backend/app/utils/prompts.py:42-125

Level 4: RAG-Backed Explanations

The ExplanationAgent combines LLM generation with retrieved document context.

1. Context Retrieval: Queries the RAG service using the prompt and response.
2. Synthesis: Uses LEVEL_4_EXPLANATION_PROMPT to create a critique based on the retrieved guidelines.
3. Source Attribution: The LLM maps its reasoning to specific doc_id and chunk_id citations.

Sources: backend/app/services/explanation_agent.py:27-58, backend/app/utils/prompts.py:157-185

Safety & Alignment Services

Beyond the main pipeline, specialized services use Gemini to ensure response quality.

Context Alignment Service

Analyzes "Intent Match" and "Topic Relevance" to detect if an AI drifted from the user's question.

# Dimensions evaluated by the LLM:
# 1. Intent Match (0.0-1.0)
# 2. Topic Relevance (0.0-1.0)
# 3. Question Coverage (Multi-part check)

Sources: backend/app/services/context_alignment_service.py:53-73

Safety Assessment Service

Evaluates the quality of advice rather than just the presence of keywords. It specifically checks for emergency handling and self-harm resource provision. It classifies interactions into safe, review, or unsafe categories based on an LLM-generated risk score.
Sources: backend/app/services/safety_assessment_service.py:75-103

Conclusion

The LLM integration is a multi-layered system that prioritizes clinical safety through conservative prompting and strict JSON schemas. By utilizing Gemini for both analysis and RAG-driven explanations, the system provides a transparent and auditable watchtower for medical AI behavior.

Sources: AI Service/plan.md:214-219, backend/app/services/gemini_service.py:148