Media Transcription (Whisper)

Relevant source files

The following files were used as context for generating this wiki page:

• apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/MediaWhisperTranscriptionService.kt
• apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/OpenAiWhisperTranscriptionClient.kt
• apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/YouTubeExtractionProvider.kt
• apps/api/src/main/kotlin/com/briefy/api/infrastructure/extraction/XApiExtractionProvider.kt
• apps/api/src/main/kotlin/com/briefy/api/application/source/SourceService.kt
• apps/api/src/test/kotlin/com/briefy/api/infrastructure/extraction/MediaWhisperTranscriptionServiceTest.kt

The Media Transcription system provides a robust pipeline for converting audio content from video platforms (such as YouTube and X/Twitter) into text using OpenAI's Whisper model. This service is a critical component of the content extraction layer, enabling the project to process non-textual media sources for further summarization and analysis.

The system handles the complexities of media processing, including downloading video/audio streams, segmenting long audio files to meet API constraints, and orchestrating the transcription of these segments back into a unified transcript.

System Architecture

The transcription architecture is built on a layered approach where platform-specific providers utilize a centralized transcription service. This service, in turn, relies on external command-line tools like ffmpeg for media manipulation and a dedicated client for Whisper API interactions.

Component Overview

Component	Description	Primary File
MediaWhisperTranscriptionService	Orchestrates the end-to-end transcription flow including audio segmentation and client calls.	MediaWhisperTranscriptionService.kt
OpenAiWhisperTranscriptionClient	Handles the direct HTTP communication with the OpenAI Whisper API.	OpenAiWhisperTranscriptionClient.kt
YouTubeExtractionProvider	Extracts metadata and downloads audio from YouTube URLs.	YouTubeExtractionProvider.kt
XApiExtractionProvider	Handles video content discovery and downloading from X (formerly Twitter).	XApiExtractionProvider.kt
ffmpeg	External dependency used for audio extraction, conversion to MP3, and segmentation.	MediaWhisperTranscriptionService.kt:14

Transcription Data Flow

The following diagram illustrates how a media file is processed from its raw state to a final transcript:

The workflow ensures that large media files are broken into 900-second (15-minute) segments to remain compatible with transcription client limits.
Sources: MediaWhisperTranscriptionService.kt:22-42, MediaWhisperTranscriptionServiceTest.kt:24-42

Core Implementation Details

Audio Segmentation and Processing

The MediaWhisperTranscriptionService uses ffmpeg to extract audio and split it into segments. It explicitly ignores video tracks (-vn) and converts audio to a low-bitrate MP3 format (libmp3lame, 64k) to optimize upload speed and storage.

// Execution of FFmpeg command for segmentation
runCommand(
    listOf(
        ffmpegPath,
        "-hide_banner",
        "-loglevel", "error",
        "-i", mediaFile.absolutePath,
        "-vn",
        "-f", "segment",
        "-segment_time", "900",
        "-c:a", "libmp3lame",
        "-b:a", "64k",
        File(segmentDir, "chunk-%03d.mp3").absolutePath
    )
)

Sources: MediaWhisperTranscriptionService.kt:27-40

Multi-Platform Integration

The transcription service is utilized by multiple extraction providers when native captions or transcripts are unavailable.

YouTube Extraction

The YouTubeExtractionProvider first attempts to retrieve subtitles. If subtitles are missing, it falls back to downloading the "bestaudio" and transcribing it via Whisper.
Sources: YouTubeExtractionProvider.kt:33-43

X (Twitter) Extraction

The XApiExtractionProvider detects video or animated GIF media in posts. It downloads the video variant with the highest bitrate (up to a configured limit) and transcribes the audio if the duration is within the maxVideoDurationSeconds limit.
Sources: XApiExtractionProvider.kt:185-235, XApiExtractionProvider.kt:509-514

Configuration and Constraints

The system relies on several environment-defined properties to control execution and prevent resource exhaustion.

Property	Default	Description
extraction.media.ffmpeg-path	ffmpeg	Path to the ffmpeg executable.
extraction.media.command-timeout-seconds	180	Maximum time allowed for any single ffmpeg command.
extraction.youtube.max-duration-seconds	7200	Maximum video length (2 hours) allowed for processing.
maxVideoDownloadBytes	-	Limit on the size of the video file downloaded from X.

Sources: MediaWhisperTranscriptionService.kt:12-16, YouTubeExtractionProvider.kt:15-20, XApiExtractionProvider.kt:27-33

Error Handling and Resiliency

The transcription process includes specific checks to ensure reliability:

• Segment Validation: The service throws an IllegalStateException if ffmpeg fails to produce any MP3 chunks.
• Process Monitoring: A ProcessOutputCollector runs in a daemon thread to capture logs from external commands, preventing pipe deadlocks and providing context on failure.
• Timeout Management: If a command exceeds the commandTimeoutSeconds, the process is forcibly destroyed, and a timeout exception is raised with a preview of the captured output.

Summary

Media Transcription (Whisper) acts as the bridge between raw audio/visual content and the text-centric processing engine of the application. By combining the power of ffmpeg for local media manipulation and OpenAI's Whisper for high-accuracy speech-to-text, the system provides a seamless experience for transcribing long-form content from diverse web platforms. Significant emphasis is placed on segmenting audio to handle large files and robust monitoring of external process execution.