Medical Document OCR
Relevant source files
The following files were used as context for generating this wiki page:
The Medical Document OCR system is a specialized AI-powered module within the OpenMRS ecosystem designed to convert physical and digital medical documents into structured, actionable data. It leverages advanced Large Language Models (LLMs) and Optical Character Recognition (OCR) engines, specifically DeepSeek-OCR and Google Gemini, to extract clinical information from PDF files and images.
This system serves as a core component of the "AI/ML Backend" layer, providing automated data entry capabilities that reduce manual transcription efforts for hospital staff. It integrates with the broader patient dashboard to allow for document-to-markdown conversion and structured data extraction, supporting clinical workflows like vitals recording and patient history documentation.
Sources: conductor/product.md:1-10, AGENTS.md:5-10, openMRS-main/DeekSeek-OCR---Dockerized-API-main/README.md:1-5
System Architecture
The OCR functionality is hosted within the AI/ML Backend, primarily utilizing a FastAPI-based REST API and Dockerized environments for model execution. The architecture is split between local processing (DeepSeek-OCR via vLLM) and cloud-based extraction (Google Gemini).
Document Processing Flow
The system supports multiple processing paths depending on the desired output format, ranging from raw text extraction to structured Markdown that preserves document layout.
The flow shows the selection between local GPU-intensive OCR and cloud-based AI extraction for medical records.
Sources: openMRS-main/DeekSeek-OCR---Dockerized-API-main/README.md:204-220, deployment/HOSPITAL-DEPLOYMENT-GUIDE.md:20-35
Core Components and Technologies
1. DeepSeek-OCR Backend
A powerful OCR solution that converts PDF documents to Markdown format. It uses a FastAPI backend and is optimized for NVIDIA GPUs with CUDA support.
| Component | Requirement / Technology | Purpose |
|---|---|---|
| GPU | NVIDIA (CUDA 11.8+) | Model acceleration |
| VRAM | Min 12GB (Model ~9GB) | DeepSeek-OCR model hosting |
| RAM | 32GB - 64GB | System memory for large PDFs |
| Inference Engine | vLLM | High-throughput model serving |
Sources: openMRS-main/DeekSeek-OCR---Dockerized-API-main/README.md:15-30, AGENTS.md:38-42
2. OCR Processors
The system provides specialized Python scripts to handle different extraction requirements:
- Standard Markdown: Converts PDFs to structured Markdown using grounding prompts. (Sources:
pdf_to_markdown_processor.py) - Enhanced Processor: Includes image extraction, special token cleanup, and layout processing. (Sources:
pdf_to_markdown_processor_enhanced.py) - Raw OCR: Plain text extraction without formatting. (Sources:
pdf_to_ocr_enhanced.py) - Custom Prompting: Uses YAML-based configurations for specialized medical extraction tasks. (Sources:
pdf_to_custom_prompt.py)
Sources: openMRS-main/DeekSeek-OCR---Dockerized-API-main/README.md:105-180
API Endpoints and Integration
The OCR service exposes several REST endpoints for document ingestion. These are typically accessed via the AI Backend (port 3001) or the dedicated DeepSeek-OCR Docker port (8000).
API Specification
| Endpoint | Method | Parameters | Description |
|---|---|---|---|
/ocr/image | POST | file (Multipart) | Process a single medical image/photo. |
/ocr/pdf | POST | file (Multipart), prompt | Convert a PDF to Markdown/Text. |
/ocr/batch | POST | files (Array) | Process multiple documents concurrently. |
/health | GET | None | Checks model status and CUDA availability. |
Sources: openMRS-main/DeekSeek-OCR---Dockerized-API-main/README.md:225-275, deployment/HOSPITAL-DEPLOYMENT-GUIDE.md:160-170
Document-to-Vitals Integration
While the OCR module handles general document conversion, it works in tandem with the vitals extraction service.
This diagram illustrates the interaction between the hospital staff and the AI backend for automated data extraction.
Sources: readmes/IMPLEMENTATION_SUMMARY.md:45-65, readmes/FINAL_SUMMARY.md:100-115
Configuration and Deployment
The OCR system is highly sensitive to hardware and API configuration. In a hospital deployment, these settings are managed through environment variables and Docker Compose.
- Prompt Engineering: The module uses specific prompts like
<image>\n<|grounding|>Convert the document to markdown.to ensure clinical accuracy. (Sources:custom_config.py) - Environment Variables:
GEMINI_API_KEY: Required for cloud-based extraction.MAX_CONCURRENCY: Controls how many documents are processed at once.GPU_MEMORY_UTILIZATION: Limits VRAM usage (default 0.85).
Sources: openMRS-main/DeekSeek-OCR---Dockerized-API-main/README.md:380-400, deployment/HOSPITAL-DEPLOYMENT-GUIDE.md:85-100
Summary
Medical Document OCR provides a critical bridge between paper-based workflows and the OpenMRS digital chart. By combining local DeepSeek-OCR capabilities for high-throughput document conversion with Google Gemini's reasoning for clinical data extraction, the system enables features such as auto-populating vitals forms and generating intelligent patient summaries. This integration is designed for high-concurrency hospital environments, requiring specific GPU hardware for local model hosting or secure cloud API access for extraction.
Sources: conductor/product.md:15-30, deployment/HOSPITAL-DEPLOYMENT-GUIDE.md:10-15