Whisper.cpp Integration
Relevant source files
The following files were used as context for generating this wiki page:
Whisper.cpp integration serves as a core component of the Crisp video cleaning pipeline, specifically responsible for high-accuracy speech transcription and filler word detection. Within the Crisp engine, whisper.cpp is utilized to generate word-level timestamps that allow the application to identify and remove hesitation words such as "um", "uh", and "hmm" from audio and video streams simultaneously. Sources: README.md:46-48, CLAUDE.md:104-106
The integration is designed to be self-contained on macOS, leveraging the whisper-cli binary built from source to ensure compatibility with Apple Silicon (arm64) hardware and Metal acceleration. While the application now offers an experimental, faster alternative called Wren, Whisper.cpp remains the primary high-fidelity backend for transcription-based filler removal. Sources: README.md:55-58, CLAUDE.md:144-147
Architecture and Data Flow
Whisper.cpp is integrated as a vendored binary that the Python-based engine shells out to during the transcription phase of the cleaning pipeline.
The diagram shows how Whisper fits into the overall detection and rendering pipeline.
Sources: README.md:43-52, CLAUDE.md:104-110, research/NOTES.md:129-134
Transcription Logic
The engine invokes whisper-cli using a pre-downloaded speech model. The process follows these specific steps:
- Audio Extraction:
ffmpegextracts a 16kHz mono WAV file from the source video, which is the required input format for Whisper.cpp. Sources: setup.sh:13-15, research/README.md:15-17 - Transcription:
whisper-cliprocesses the WAV file to produce word-level timestamps. Sources: README.md:46-48 - Filler Matching: The engine matches the transcribed words against a vocabulary of hesitation sounds (e.g., "uh", "um"). Sources: CLAUDE.md:104-106
- Span Identification: These timestamps are converted into time ranges for the
ffmpegre-render phase. Sources: README.md:49-52
Component Management
Crisp manages Whisper.cpp through a combination of automated setup scripts and build-time vendoring to ensure the app is self-contained.
Binary Vendoring
Because no official macOS CLI binary is published for Whisper.cpp, Crisp builds whisper-cli from a pinned Git tag during the build process.
| Component | Source / Version | Build Method | Purpose |
|---|---|---|---|
whisper-cli | v1.9.0 | cmake | Core transcription engine |
ggml-base.en.bin | Hugging Face | curl (downloaded) | Speech model weights (~148MB) |
Sources: apps/desktop/Scripts/vendor.sh:22-25, 68-80, setup.sh:13-15
Build and Deployment
The vendor.sh script handles the compilation of whisper-cli with specific optimizations for Apple Silicon, including -DGGML_METAL=ON for hardware acceleration. Sources: apps/desktop/Scripts/vendor.sh:75-78
# Example build command used in vendor.sh
cmake -S "$SRC" -B "$SRC/build" -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_OSX_ARCHITECTURES=arm64 -DBUILD_SHARED_LIBS=OFF \
-DWHISPER_BUILD_TESTS=OFF -DWHISPER_BUILD_EXAMPLES=ON \
-DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON
Sources: apps/desktop/Scripts/vendor.sh:75-78
Model Lifecycle
Unlike the binary, the speech model is not bundled with the application to prevent bloat and unnecessary re-shipping during updates. Sources: CLAUDE.md:153-156
- First-Run Download: The
ModelStoreservice in the Swift application fetchesggml-base.en.binfrom Hugging Face upon first use. Sources: CLAUDE.md:156-158, setup.sh:38-42 - Storage: The model is stored in the channel-specific data directory (e.g.,
~/.crisp*/models/). Sources: CLAUDE.md:157 - Verification: The download process includes SHA-256 verification and atomic publishing to ensure model integrity. Sources: CLAUDE.md:158
- Path Resolution: The Swift layer sets the
CRISP_WHISPERenvironment variable to point to the bundled binary, allowing the engine to locate it regardless of the user's PATH. Sources: CLAUDE.md:148-151
Performance and Limitations
Whisper.cpp provides high accuracy but involves trade-offs compared to the experimental Core ML backend.
- Accuracy: Whisper is more conservative than the custom Wren model; it only cuts sounds that it explicitly transcribes as filler words, which helps maintain the flow of natural delivery. Sources: research/NOTES.md:175-180
- Resource Usage: Running Whisper.cpp typically causes system fans to ramp up significantly due to the heavy transcription load. Sources: research/NOTES.md:188-189
- Captions: Caption generation (SRT/VTT) is a Whisper-only feature. If the user enables the fast filler model (Wren), captions are automatically disabled because the custom model lacks transcription capabilities. Sources: research/NOTES.md:310-318
Whisper.cpp remains the fallback and high-accuracy standard within the Crisp ecosystem, ensuring that users have a reliable, albeit slower, option for clean video production. Sources: research/NOTES.md:129-134, CLAUDE.md:107-110