Gemini AI Information Extraction
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Gemini AI Information Extraction is a core component of the OpenMRS AI-assisted EHR system, designed to transform unstructured medical data into structured, actionable clinical information. By leveraging Google’s Gemini 2.5 Flash and 2.0 Flash models, the system can parse natural language transcriptions of patient vitals and perform Optical Character Recognition (OCR) on medical documents and PDFs.
The primary scope of this module includes the extraction of ten critical vital sign fields, automatic unit conversion (e.g., Fahrenheit to Celsius), and the structural formatting of data for auto-populating OpenMRS forms. This significantly reduces manual data entry for clinical staff and improves data accuracy through intelligent parsing and calculation.
Sources: readmes/FINAL_SUMMARY.md, openMRS-main/vitals_gemini_test.py:15-18
System Architecture and Data Flow
The extraction system operates within the Python-based AI Backend, primarily interacting with the Gemini API to process text and image inputs. It serves as a middleware layer between raw inputs (voice transcriptions or images) and the OpenMRS frontend.
Information Extraction Pipeline
The following diagram illustrates how raw medical data is processed through the Gemini extraction layer to reach the structured EHR database.
Vitals Extraction Logic
The vitals extraction module utilizes the gemini-2.5-flash model to identify clinical parameters within natural language text. The system employs a two-step process: raw extraction and unit normalization.
Core Processing Steps
- Lazy Initialization: The system initializes the Gemini API only when needed to handle potential import errors gracefully.
- LLM Prompting: A structured prompt instructs the model to identify numerical values and units for temperature, blood pressure, heart rate, respiration, SpO2, weight, height, MUAC, and notes.
- Regex Parsing: The raw LLM response is parsed using regular expressions to isolate numerical values from their associated unit strings.
- Unit Conversion: The system automatically standardizes values into clinical units (Celsius, kg, cm) regardless of the input format (Fahrenheit, lbs, inches).
Sources: openMRS-main/vitals_gemini_test.py:53-75, openMRS-main/vitals_gemini_test.py:116-160
Extracted Data Model
The system targets the following structured data fields:
| Field | Target Unit | Supported Input Units |
|---|---|---|
| Temperature | Celsius (°C) | F, Fahrenheit, C, Celsius |
| Blood Pressure | mmHg (systolic/diastolic) | "120/80", "120 over 80" |
| Heart Rate | beats/min | bpm, beats per minute |
| Respiration Rate | breaths/min | rpm, breaths per minute |
| SpO2 | % | percentage |
| Weight | kg | lbs, pounds, kg, kilograms |
| Height | cm | inches, in, cm, feet, ft |
| BMI | kg/m² | Auto-calculated from Weight/Height |
| Sources: openMRS-main/vitals_gemini_test.py:35-51, readmes/IMPLEMENTATION_SUMMARY.md |
Gemini OCR System
The OCR sub-system uses the gemini-2.0-flash model to process image-based documents and PDFs. It includes specialized pre-processing to ensure images fit within model constraints.
Image and PDF Processing Flow
The sequence below details the interaction between the system and the Gemini API during document extraction.
Key OCR Functions
resize_image_for_ocr: Resizes images to a maximum dimension (default 1024px) while preserving aspect ratio and converting to RGB/JPEG/PNG as required.cleanup_ocr_output: A post-processing utility that removes LLM-specific special tokens (e.g.,<|ref|>,<|det|>,<image>) and cleans up redundant whitespace.ocr_pdf_pages_via_gemini: Renders PDF pages into PNG images usingpymupdfbefore iterating through them for individual OCR processing.
Sources: openMRS-main/ocr_gemini.py:32, openMRS-main/ocr_gemini.py:63, openMRS-main/ocr_gemini.py:186
Configuration and Implementation
Extraction is configured through environment variables to manage API keys and model selection.
Configuration Options
| Variable | Default Value | Description |
|---|---|---|
GEMINI_API_KEY | (Required) | API Key from Google AI Studio |
GEMINI_OCR_MODEL | gemini-2.5-flash | The model used for OCR tasks |
timeout_seconds | 120 | Maximum time allowed for API response |
max_retries | 2 | Number of retry attempts on failure |
| Sources: openMRS-main/ocr_gemini.py:15-28, readmes/VOICE_VITALS_QUICK_START.md |
Implementation Example: Vitals Extraction
# From openMRS-main/vitals_gemini_test.py
def fahrenheit_to_celsius(fahrenheit):
return (fahrenheit - 32) * 5/9
def calculate_bmi(weight_kg, height_cm):
if height_cm == 0: return 0
height_m = height_cm / 100
return weight_kg / (height_m ** 2)
Sources: openMRS-main/vitals_gemini_test.py:24-33
Conclusion
Gemini AI Information Extraction provides a robust bridge between unstructured clinical inputs and the OpenMRS data model. By combining LLM-based natural language understanding with sophisticated OCR processing, the system automates the entry of vitals and document data. Its internal unit conversion and data normalization logic ensure that clinicians interact with standardized data regardless of the source format, enhancing both the efficiency and reliability of the EHR.
Sources: readmes/FINAL_SUMMARY.md, openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md