Audio Capture Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Audio Capture Pipeline in MacParakeet is a sophisticated multi-stream system designed to handle real-time audio input for three primary use cases: system-wide dictation, file/URL transcription, and meeting recording. It utilizes high-performance local speech recognition via Parakeet on the Apple Neural Engine (ANE) to provide fast, private, on-device processing.
The pipeline architecture prioritizes local-first data posture, ensuring that audio and transcripts remain on the device. For meeting scenarios, the pipeline supports concurrent dual-stream capture of both microphone and system audio (via ScreenCaptureKit), aligned by host time to facilitate multi-speaker diarization and accurate meeting artifacts.
Sources: spec/05-audio-pipeline.md:1-10, AGENTS.md:9-15, spec/00-vision.md:13-25
Pipeline Architecture & Components
The audio pipeline is structured around a shared engine model to manage hardware resources efficiently, especially when multiple capture sessions (like simultaneous dictation and meeting recording) occur.
Core Components
| Component | Responsibility |
|---|---|
| SharedMicrophoneStream | Manages a process-wide AVAudioEngine instance; handles fan-out to multiple subscribers (Dictation, Meeting Mic). |
| SystemAudioStream | Wraps ScreenCaptureKit SCStream to capture system-level audio without requiring HAL aggregate devices. |
| MeetingAudioPairJoiner | Synchronizes dual streams (Mic/System) using host time and handles buffering for live chunking. |
| CaptureOrchestrator | Manages the high-level ingest, offset alignment, and chunking flow for live previews. |
| MicrophoneCapture | Acts as a subscriber to the shared stream with specific policies (Raw vs. Voice Processing). |
Sources: spec/05-audio-pipeline.md:195-215, spec/05-audio-pipeline.md:119-140
Audio Data Flow (Meeting Recording)
The following diagram illustrates the flow from hardware/system capture to the final processed artifact during a meeting recording session.
Microphone Capture (Dictation)
Dictation utilizes the SharedMicrophoneStream with a policy typically set to wantsVPIO: false to ensure raw, high-quality audio for the Parakeet engine.
Instant Dictation and Pre-roll
A key feature is Instant Dictation, which maintains a passive subscriber to the shared stream even while idle. This subscriber stores a bounded 1-second RAM-only ring buffer at 16kHz mono. When the user triggers dictation, up to 0.45 seconds of this pre-roll is prepended to the new recording to ensure the start of speech isn't truncated.
- Format: 16kHz mono Float32 WAV.
- Threshold: A minimum of 4,800 samples (0.3s) is required for a valid recording.
- Media Interaction: If system media is playing, the pipeline can optionally trim the pre-roll to avoid capturing speaker bleed from other apps.
Sources: spec/05-audio-pipeline.md:10-35
Meeting Recording Pipeline (v0.6)
Meeting recording introduces dual-stream complexity, requiring the capture of both the user (Microphone) and other participants (System Audio).
Dual-Stream Capture and Alignment
System audio is captured via ScreenCaptureKit to avoid the instability of managing virtual audio drivers. Microphone audio is pulled from the shared engine. Both streams are timestamped using host time to ensure they can be joined accurately.
Echo Mitigation and AEC
MacParakeet implements multiple layers to handle acoustic echo cancellation (AEC) during meetings:
- Voice Processing I/O (VPIO): The shared engine can be promoted to VPIO mode, which engages Apple's hardware-accelerated echo cancellation.
- Software AEC: A joined software-based AEC (NLMS) is used in the
MeetingRecordingServiceto process mic/system pairs. - Cross-Stream Energy Gate: A deterministic guard monitors Root Mean Square (RMS) energy. If system audio energy strongly dominates mic energy within a specific window, the mic chunk is suppressed for live transcription to prevent phantom "Me" attributions.
Sources: plans/completed/2026-04-meeting-mic-aec.md:11-25, spec/05-audio-pipeline.md:144-165
Storage and Recovery
Audio is stored as fragmented M4A files (AAC 64kbps, 48kHz mono) for each source. This fragmentation ensures that if the app crashes, the recording remains playable up to the last 1-second fragment.
| Artifact | Description |
|---|---|
microphone.m4a | Raw or VPIO-processed mic audio. |
system.m4a | System audio captured via ScreenCaptureKit. |
meeting.m4a | Final stereo mix (L=Mic, R=System). |
recording.lock | Recovery state file containing session notes and STT engine info. |
Sources: spec/05-audio-pipeline.md:175-190, spec/05-audio-pipeline.md:119-140
Stream Synchronization (MeetingAudioPairJoiner)
The MeetingAudioPairJoiner is responsible for aligning the disparate Mic and System audio buffers.
Synchronization Logic
The joiner maintains internal queues for both streams and uses Host Time (mach_absolute_time) to determine the offset. It handles "drift" and ensures that chunks sent for live transcription are time-aligned.
Audio Format Specifications
All audio ultimately reaches the STT engine in a normalized format optimized for the Apple Neural Engine.
| Pipeline Stage | Sample Rate | Channels | Bit Depth | Format |
|---|---|---|---|---|
| Hardware Input | Native | Varies | Float32 | PCM |
| STT Input | 16kHz | Mono | Float32 | WAV |
| Meeting Storage | 48kHz | Mono | 64kbps | M4A (AAC) |
| Final Meeting Mix | 48kHz | Stereo | 128kbps | M4A (AAC) |
Sources: spec/05-audio-pipeline.md:270-285
Conclusion
The Audio Capture Pipeline serves as the foundation for MacParakeet's local-first voice features. By abstracting hardware management into the SharedMicrophoneStream and implementing robust synchronization logic via the MeetingAudioPairJoiner, the system achieves high reliability and performance across dictation and multi-stream meeting scenarios. This architecture allows the app to maintain professional-grade audio artifacts while providing real-time, privacy-focused transcription capabilities.