Transcription Workflow
Relevant source files
The following files were used as context for generating this wiki page:
The Transcription Workflow is a multi-stage pipeline responsible for converting audio recordings into structured, speaker-labeled text. It leverages a hybrid approach involving real-time processing via WebSockets for the web application and asynchronous background tasks via Celery for heavy-duty processing. The system utilizes OpenAI's Whisper model for speech-to-text and Pyannote for speaker diarization to ensure high-quality, attributed transcripts.
This workflow integrates closely with the AI Notes Generation system to provide summarized insights and action items once transcription is complete. It supports session-aware self-correction, noise reduction, and term consistency across audio chunks.
Sources: ai_backend/services/enhanced_transcription_service.py:1-40, README.md:14-25, web_app/src/main.js:1-15
System Architecture
The transcription architecture is divided into the ingestion layer (Web/Mobile), the processing layer (FastAPI & Celery), and the persistence layer (PostgreSQL & Redis).
The diagram shows the flow of audio data from the client through the backend services to the machine learning models and final storage.
Sources: README.md:14-25, ai_workers/transcription_worker.py:25-35
Audio Ingestion and Chunking
Audio is captured and processed in chunks to support scalability and long-form recordings. The web application captures audio in 1-second intervals using the MediaRecorder API and transmits it to the backend.
Web Audio Capture
The ClassMateWebApp class handles microphone access and streaming. It uses getUserMedia with specific constraints for optimal transcription quality:
- Sample Rate: 16000 Hz
- Channels: Mono (handled during preprocessing)
- Features: Echo cancellation and noise suppression enabled
Sources: web_app/src/main.js:113-132, ai_backend/services/enhanced_transcription_service.py:34-36
Public API Ingestion
The public REST API provides a /transcribe endpoint that accepts an audio_url or audio_data. Requests are validated and then offloaded to the Celery worker via process_transcription_task.delay().
| Parameter | Type | Description |
|---|---|---|
session_id | String | Unique identifier for the recording session |
chunk_index | Integer | Order of the chunk in the sequence |
audio_file_path | String | Path to the temporary audio file on disk |
language | String (Optional) | ISO language code for transcription |
Sources: ai_backend/api/public_api.py:101-125, ai_backend/shared_contracts/models.py:28-33
Enhanced Processing Pipeline
The EnhancedTranscriptionService manages the core logic of converting raw audio into refined text. It follows a strictly defined pipeline of preprocessing, inference, and post-processing.
1. Preprocessing and Noise Reduction
Before transcription, the service performs the following steps:
- Resampling: Ensures audio is at 16kHz.
- Spectral Subtraction: Uses
librosato estimate noise from the first 0.5 seconds of the chunk and applies a spectral mask to reduce background noise. - Normalization: Adjusts audio to a consistent -20 dBFS level.
Sources: ai_backend/services/enhanced_transcription_service.py:206-250
2. Machine Learning Inference
The system uses whisper.load_model("large-v3") for high-accuracy speech-to-text. It supports:
- Language Detection: Auto-detects the spoken language if not provided.
- Word Timestamps: Generates precise start/end times for every word.
- Diarization: Uses
pyannote/speaker-diarization-3.1to label different speakers (e.g., "SPEAKER_01").
Sources: ai_backend/services/enhanced_transcription_service.py:53-75, ai_backend/services/enhanced_transcription_service.py:89-115
3. Post-Transcription Chain
After the raw text is generated, a multi-agent chain refines the output:
The agents perform specific duties to improve readability and accuracy.
Sources: ai_backend/agents/transcription_post_agents.py:93-110
| Agent | Responsibility |
|---|---|
| CleanupAgent | Normalizes whitespace and fixes punctuation spacing. |
| TermConsistencyAgent | Applies session-learned replacements for technical terms. |
| ConfidenceFilterAgent | Flags low-confidence segments (avg_logprob < -1.5). |
| SpeakerTurnAgent | Prefixes text with speaker labels (e.g., "SPEAKER_01: Hello"). |
Sources: ai_backend/agents/transcription_post_agents.py:15-85
Data Persistence and Session Memory
The TranscriptionWorker ensures results are saved to the database and session metadata is updated.
Database Schema (Simplified)
- transcript_chunks: Stores individual chunk results, including
text,confidence,speaker, andchunk_index. - sessions: Tracks progress via
uploaded_chunksand updatesstatusto "completed" once all chunks are processed.
Sources: ai_workers/transcription_worker.py:65-90
Session-Aware Learning
The service utilizes a SQLiteMemoryStore to maintain "Session Memory." This allows the system to learn patterns within a single lecture:
- Casing Map: Learns the preferred casing for terms used frequently in a session.
- Term Frequency: Tracks top terms to provide context for future chunks.
- Self-Correction: Automatically replaces misrecognized words based on previously learned correct terms in the same session.
Sources: ai_backend/services/enhanced_transcription_service.py:146-193
Sequence of Operations
The following sequence diagram illustrates the complete flow from an audio chunk upload to the final database update.
Summary
The Transcription Workflow provides a robust, session-aware pipeline for transforming audio into high-fidelity text. By combining advanced noise reduction, large-scale ML models, and a post-processing agent chain, it ensures that transcripts are not only accurate but also formatted for immediate use in study materials and notes. Its distributed nature via Celery allows the system to scale horizontally to handle high-concurrency university environments.
Sources: README.md:1-10, GAPS_IMPLEMENTATION_SUMMARY.md:380-400