Sarvam AI STT Integration
Relevant source files
The following files were used as context for generating this wiki page:
- openMRS-main/BATCH_API_INTEGRATION.md
- readmes/FINAL_SUMMARY.md
- readmes/IMPLEMENTATION_SUMMARY.md
- readmes/VOICE_VITALS_QUICK_START.md
- openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/vitals-recording.service.ts
- openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md
Introduction
The Sarvam AI Speech-to-Text (STT) integration is a core component of the Voice Vitals Recording feature within the OpenMRS ecosystem. It enables clinical staff, such as nurses and doctors, to record verbal patient vitals and convert them into structured text for automated form population. This integration bridges the gap between natural voice communication and the structured data requirements of an Electronic Health Record (EHR) system.
Sources: readmes/FINAL_SUMMARY.md:1-5, readmes/README_VOICE_VITALS.md:1-10
The integration operates as a middleware service within the AI backend, processing audio blobs captured by the frontend Web Audio API. It handles diverse audio lengths through a dual-path processing logic that selects between synchronous and asynchronous Sarvam APIs based on recording duration.
Sources: openMRS-main/BATCH_API_INTEGRATION.md:1-10, openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/vitals-recording.service.ts
System Architecture and Data Flow
The integration architecture consists of a React-based frontend that captures audio and a FastAPI-based backend that interfaces with Sarvam AI's cloud services.
Logic Flow for Voice-to-Text
The system follows a specific sequence from audio capture to text extraction:
- Audio Capture: The
useVitalsRecorder.tshook captures audio via the Web Audio API. - Preprocessing: The frontend transmits the audio blob to the
/transcribe_batchor/transcribe_asyncendpoint. - Transcription Selection: The backend routes the request to Sarvam's Regular or Batch STT API based on the 30-second threshold.
- Extraction: The resulting transcript is then passed to the Gemini API for structured vital extraction.
Sources: readmes/IMPLEMENTATION_SUMMARY.md:38-55, openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md:94-135
Dual-Path Processing Diagram
The following diagram illustrates how the system branches based on recording length:
Functional Components
Frontend Service Layer
The vitals-recording.service.ts acts as the primary orchestrator for transcription requests. It manages the lifecycle of the recording transmission and handles errors such as network timeouts or API failures.
| Feature | Description |
|---|---|
transcribeAudioRecording() | Sends audio blobs to the backend endpoint. |
recordAndExtractVitals() | Orchestrates the full pipeline from transcription to extraction. |
| Automatic Format Handling | Converts WebM (frontend capture) to WAV for Sarvam compliance. |
Sources: readmes/IMPLEMENTATION_SUMMARY.md:18-28, openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md:43-55
Backend API Endpoints
The backend, implemented in main.py, exposes specific endpoints for Sarvam integration:
| Endpoint | Method | Purpose |
|---|---|---|
/transcribe_batch | POST | Handles short recordings (≤30s) using the saaras:v2.5 model. |
/transcribe_async | POST | Initiates a batch job for long recordings using the saarika:v2.5 model. |
/transcription_status/{job_id} | GET | Polls for the status and result of asynchronous batch jobs. |
Sources: openMRS-main/BATCH_API_INTEGRATION.md:25-50, openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md:69-80
API Configuration and Models
Successful integration requires specific environment variables and model identifiers to communicate with Sarvam AI's infrastructure.
Model Specification
Sarvam AI provides different models optimized for immediate response versus high-throughput batch processing:
- Regular API Model:
saaras:v2.5(Optimized for latency, <30s). - Batch API Model:
saarika:v2.5(Optimized for accuracy and long durations).
Sources: openMRS-main/BATCH_API_INTEGRATION.md:90-95
Environment Variables
The system relies on the following secrets for authentication:
# Sarvam API Key for authentication
SARVAM_API_KEY=your_sarvam_key_here
# Base URL for Sarvam STT services
SARVAM_STT_URL=https://api.sarvam.ai/v1/speech-to-text
Sources: readmes/VOICE_VITALS_QUICK_START.md:5-15, openMRS-main/BATCH_API_INTEGRATION.md:85-89
Sequence Diagram: Asynchronous Transcription
For recordings exceeding 30 seconds, the system utilizes an asynchronous polling mechanism to ensure reliable transcription of large audio files.
Error Handling and Performance
The integration includes robust error handling to manage the complexities of cloud-based voice processing.
Typical Latency Metrics
- Short Recordings (10-30s): 5–15 seconds total processing time.
- Medium Recordings (1-2 min): 1–2 minutes total processing time.
- Long Recordings (5-10 min): 3–5 minutes total processing time.
Sources: openMRS-main/BATCH_API_INTEGRATION.md:78-83
Error Scenarios
The system provides user-friendly notifications for the following failure states:
- Microphone Access: Handled in the frontend hook if permissions are denied.
- Transcription Timeout: Long recordings are capped with a 5-minute timeout window.
- API Key Missing: The backend returns a specific error if
SARVAM_API_KEYis not detected in the environment.
Sources: openMRS-main/BATCH_API_INTEGRATION.md:70-77, openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md:175-185
Conclusion
The Sarvam AI STT integration serves as the critical first step in the Voice Vitals pipeline, converting raw clinical speech into the text required for AI-driven data extraction. By supporting both real-time and batch processing, it ensures that medical staff can record observations of any length while maintaining a high degree of transcription accuracy through specialized models like saaras and saarika.