Audio Processing & Transcription
Relevant source files
The following files were used as context for generating this wiki page:
The Audio Processing & Transcription system in ThinkEx is designed to transform raw audio files into structured, AI-readable knowledge. It provides a seamless pipeline from audio ingestion and security validation to durable transcription workflows powered by Google’s Gemini models. Within the workspace, audio is treated as a first-class media type, enabling users to interact with synchronized transcripts, speaker-identified timelines, and automated summaries.
Sources: src/app/api/audio/process/route.ts:10-15, README.md:38-40
System Architecture & Data Flow
The audio processing lifecycle involves three main phases: request validation, durable workflow execution, and frontend state management.
1. API Ingestion & Security
The entry point for audio processing is the POST /api/audio/process endpoint. This route performs critical security checks, specifically validating the fileUrl origin to prevent Server-Side Request Forgery (SSRF). It only allows URLs from verified hosts such as Supabase storage, the application's own URL, or local storage during development.
Sources: src/app/api/audio/process/route.ts:46-70
2. Durable Transcription Workflow
Once validated, the system triggers the audioTranscribeWorkflow. This workflow utilizes Google's Gemini 2.5 Flash model to perform multi-modal analysis on the audio file. Unlike traditional Speech-to-Text (STT), this process extracts semantic meaning alongside the text.
The following diagram illustrates the transcription request flow:
The workflow returns a structured TranscribeResult containing a summary, identified segments (with speaker tags and timestamps), and the total duration.
Sources: src/workflows/audio-transcribe/steps/transcribe.ts:16-84, src/app/api/audio/process/route.ts:80-90
Audio Data Model
Audio information is stored within the AudioData structure as part of a workspace item. This structure is used both for rendering the UI and for providing context to the AI assistant.
| Field | Type | Description |
|---|---|---|
processingStatus | string | Current state: uploading, processing, failed, or complete. |
filename | string | The original name of the uploaded file. |
duration | number | Total audio length in seconds. |
summary | string | An AI-generated overview of the audio content. |
segments | AudioSegment[] | List of objects containing speaker, timestamp, content, and optional translation. |
Sources: src/lib/utils/format-workspace-context.ts:515-545, src/workflows/audio-transcribe/steps/transcribe.ts:5-10
Frontend Components & User Interface
Audio Card Management
The AudioCardContent component manages the lifecycle of the audio item on the canvas. It reactively updates based on the processingStatus.
- Processing States: While transcribing, it displays a loading state with a
Micicon and a "Analyzing audio..." message. - Error Handling: If processing fails, it provides an error message and a retry button that re-triggers the
startAudioProcessingutility. - Complete State: Once finished, it renders a custom audio player with playback speed controls (1x/2x), a seekable waveform, and a scrollable timeline.
Sources: src/components/workspace-canvas/AudioCardContent.tsx:75-120, src/components/workspace-canvas/AudioCardContent.tsx:135-150
Live Recording Visualization
During active audio recording (e.g., via the workspace microphone), the AudioWaveform component provides real-time visual feedback. It uses the Web Audio API's AnalyserNode to process frequency domain data and render it onto a canvas.
The waveform logic applies a smoothing factor (0.15) and a decay (0.92) to ensure the bars react naturally to speech patterns rather than flickering.
Sources: src/components/workspace-canvas/AudioWaveform.tsx:45-100
Integration with AI Assistant
The transcription results are formatted as part of the workspace context to allow the AI assistant to "listen" to the audio. When an audio item is selected, the formatAudioDetailsFull function generates a text-based timeline for the model.
// Example of formatted context sent to AI
if (data.segments && data.segments.length > 0) {
lines.push(` - Timeline (${data.segments.length} segments):`);
for (const seg of data.segments) {
lines.push(
` [${seg.timestamp}] ${seg.speaker}: ${seg.content}`
);
}
}
This enables the AI to answer specific questions like "What did Speaker 2 say about the project timeline?" by referencing the timestamped segments.
Sources: src/lib/utils/format-workspace-context.ts:528-545
Summary
The Audio Processing & Transcription module provides a robust foundation for integrating voice and recorded media into the ThinkEx workspace. By combining secure API routes, durable background workflows using Gemini 2.5 Flash, and interactive frontend components like the synchronized timeline and live waveforms, the system effectively turns ephemeral audio into a persistent and searchable knowledge asset.
Sources: README.md:28-32, src/components/workspace-canvas/AudioCardContent.tsx:400-420