Voice Vitals Recording Feature

Relevant source files

The following files were used as context for generating this wiki page:

• readmes/IMPLEMENTATION_SUMMARY.md
• openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md
• readmes/ARCHITECTURE_OVERVIEW.md
• readmes/README_VOICE_VITALS.md
• readmes/FINAL_SUMMARY.md
• readmes/COMPLETION_CHECKLIST.md

The Voice Vitals Recording Feature is an AI-powered system integrated into the OpenMRS Patient Chart designed to streamline clinical documentation. It allows medical staff, such as nurses and physicians, to record spoken vital signs, which are then automatically transcribed, parsed, and mapped to structured fields within the vitals biometrics form. This feature aims to reduce manual data entry and improve efficiency in busy clinical environments.
Sources: readmes/IMPLEMENTATION_SUMMARY.md, conductor/product.md

The system utilizes the Web Audio API for recording, Sarvam AI for speech-to-text transcription, and the Google Gemini API for natural language processing and vital sign extraction. It handles complex tasks such as unit conversion and blood pressure parsing, while maintaining full user control by allowing manual edits to all auto-populated fields.
Sources: readmes/README_VOICE_VITALS.md, readmes/ARCHITECTURE_OVERVIEW.md

System Architecture

The feature follows a modular architecture split between a React/TypeScript frontend and a Python/FastAPI backend. The frontend handles user interaction and audio capture, while the backend orchestrates the AI processing services.

Core Frontend Components

Component	File Path	Description
useVitalsRecorder	src/vitals/useVitalsRecorder.ts	Custom React hook for microphone access and audio recording using Web Audio API.
VitalsRecordingWidget	src/vitals/vitals-recording.component.tsx	The primary UI element providing controls (Record, Stop, Extract) and status feedback.
VitalsRecordingService	src/vitals/vitals-recording.service.ts	API communication layer handling requests to transcription and extraction endpoints.
VitalsDataMapper	src/vitals/vitals-data.mapper.ts	Utility to transform raw AI JSON responses into the react-hook-form schema format.

Sources: readmes/IMPLEMENTATION_SUMMARY.md:6-33, readmes/ARCHITECTURE_OVERVIEW.md:158-180

Logic and Data Flow

The following diagram illustrates the end-to-end data flow from the moment a nurse begins speaking to the final form submission in OpenMRS.

Backend Endpoints

The system relies on two primary backend endpoints hosted on the FastAPI server (typically port 8000).

1. Audio Transcription (/transcribe_batch)

This endpoint receives a WebM audio blob and converts it to WAV format before routing it to the Sarvam AI API. It uses a "smart routing" logic: regular API calls for recordings under 30 seconds and the batch SDK for longer durations.
Sources: readmes/COMPLETION_CHECKLIST.md:120-130, openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md:68-100

2. Vitals Processing (/process_vitals_transcription)

This endpoint accepts the transcription text and utilizes the Gemini 2.5 Flash model to extract vital signs.
Sources: readmes/IMPLEMENTATION_SUMMARY.md:46-52, readmes/ARCHITECTURE_OVERVIEW.md:106-118

Parameter	Type	Description
transcription	string	The raw text from the speech-to-text service.
Return: vitals	object	Structured JSON containing fields like Temperature, Heart Rate, etc.

Data Processing and Mapping

AI Extraction and Unit Conversion

The backend processing logic (vitals_gemini_test.py) includes robust normalization to ensure data consistency. It identifies raw values and original units, then performs the following conversions:

• Temperature: Fahrenheit (°F) to Celsius (°C)
• Weight: Pounds (lbs) to Kilograms (kg)
• Height: Inches (in) or Feet (ft) to Centimeters (cm)
• BMI: Automatically calculated from weight and height if not explicitly provided.

Sources: openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md:200-220, readmes/IMPLEMENTATION_SUMMARY.md:60-70

Form Integration Logic

The VitalsDataMapper ensures that only valid, non-null data is pushed to the form.

// Example mapping logic derived from service description
const handleVitalsExtracted = useCallback((vitalsData) => {
  const mappedData = mapVitalsToFormData(vitalsData);
  Object.entries(mappedData).forEach(([fieldName, value]) => {
    if (value !== undefined && value !== null) {
      setValue(fieldName, value); // Standard react-hook-form setter
    }
  });
}, [setValue]);

Sources: openmrs-esm-patient-chart-main/packages/esm-patient-vitals-app/src/vitals/VOICE_VITALS_IMPLEMENTATION.md:150-165, readmes/IMPLEMENTATION_SUMMARY.md:37-43

Technical Sequence Diagram

The interaction between the user interface, the recording hook, and the various AI services is detailed below.

Configuration and Deployment

The feature requires specific environment variables on the backend to function correctly.

Variable	Description
SARVAM_API_KEY	Authentication key for Sarvam AI STT services.
GEMINI_API_KEY	Authentication key for Google Gemini NLP services.
OPENMRS_API_URL	Base URL for the OpenMRS REST API.
BACKEND_URL	Frontend config pointing to the FastAPI server (default http://localhost:8000).

Sources: readmes/VOICE_VITALS_QUICK_START.md:5-15, readmes/IMPLEMENTATION_SUMMARY.md:175-185

Conclusion

The Voice Vitals Recording Feature provides a sophisticated integration of modern AI services into the OpenMRS ecosystem. By leveraging the Web Audio API for high-quality capture and advanced LLMs for clinical data extraction, the system transforms unstructured voice input into actionable medical data. The implementation prioritizes clinical accuracy through automatic unit conversions and preserves documentation integrity by maintaining full manual editability and standard form validation.
Sources: readmes/README_VOICE_VITALS.md, readmes/FINAL_SUMMARY.md