Filler Word Detection
Relevant source files
The following files were used as context for generating this wiki page:
Filler Word Detection is a core feature of the Crisp macOS application designed to identify and remove disfluencies such as "um", "uh", "hmm", and "aww" from audio and video recordings. This process enables the production of tight jump-cuts by automatically cleaning up speech patterns that distract from the main content. Sources: README.md, CLAUDE.md
The system utilizes two primary backends for detection: a transcription-based approach using whisper.cpp and a custom, lightweight, on-device machine learning model family named Wren. While whisper.cpp provides word-level timestamps through full speech-to-text, Wren offers a significantly faster (~600× real-time) alternative specifically optimized for detecting filler sounds without the overhead of full transcription. Sources: research/filler_classifier/MODEL_CARD.md, research/NOTES.md:16-20
Detection Architecture
The detection logic is split between a Python-based engine that orchestrates the cleaning pipeline and a native Swift helper that executes high-performance ML inference on macOS.
System Data Flow
The following diagram illustrates how audio is processed through the different backends to identify filler word spans.
The system identifies silence separately via FFmpeg silencedetect, allowing the filler models to focus exclusively on speech disfluencies. Sources: CLAUDE.md, research/filler_classifier/MODEL_CARD.md
Custom ML Models (Wren)
Crisp utilizes a specialized family of CNN (Convolutional Neural Network) models for audio classification. These models are optimized for the Apple Neural Engine, GPU, or CPU via Core ML.
Model Specifications
The primary model, Wren, is designed to be extremely lightweight and fast.
| Property | Wren v1 (Chunk) | Wren v2 (Sequence) |
|---|---|---|
| Parameters | ~75k | ~129k |
| Architecture | 2-D CNN (Conv-BN-ReLU) | Dilated TCN (Temporal Conv Net) |
| Input Shape | [1, 1, 64, 25] (250ms chunk) | [1, 64, T] (Full recording) |
| Output | filler_prob (0-1) | removable_prob (Per-frame) |
| Normalization | Fixed Mean/Std | Fixed Mean/Std |
| Target | "Is this a filler?" | "Is this a removable filler?" |
Sources: research/filler_classifier/MODEL_CARD.md, research/NOTES.md:200-215, apps/desktop/Sources/crisp-filler/main.swift:14-30
Audio Preprocessing
The crisp-filler native helper implements a frontend that matches torchaudio MelSpectrogram exactly. Since standard vDSP FFT requires power-of-two lengths and the system uses n_fft=400, the Discrete Fourier Transform (DFT) is implemented as a BLAS matrix multiply.
// DFT basis computation in main.swift
for nn in 0..<n {
for k in 0..<freqs {
let a = 2.0 * Double.pi * Double(k) * Double(nn) / Double(n)
c[nn * freqs + k] = Float(cos(a))
s[nn * freqs + k] = Float(sin(a))
}
}
Sources: apps/desktop/Sources/crisp-filler/main.swift:95-103
Transcription-Based Detection (Whisper)
When the custom ML model is disabled, Crisp relies on whisper.cpp to transcribe the audio. The engine then filters the resulting transcript against a specific vocabulary of filler words.
Normalization and Filtering
The Python engine normalizes words by stripping case and punctuation before comparing them to the filler vocabulary. This allows it to catch variants like "Um," or "UH!".
| Feature | Description |
|---|---|
| Normalization | Lowercasing, stripping leading/trailing spaces and punctuation. |
| Vocabulary | um, uh, uhm, er, erm, hm, hmm, mm, aww, ahem. |
| Pattern Matching | Regex shapes to catch elongated variants (e.g., "ummm", "errr"). |
Sources: apps/desktop/Resources/engine/tests/test_text.py:10-35
Logic and Post-Processing
Detection is only the first step; the system applies several gates to ensure high precision and natural-sounding cuts.
Silence Gating and Thresholding
Wren models use a default operating threshold of 0.85 to favor precision over recall, as false positives can result in cutting real speech.
- Min Filler Length: Fillers shorter than 0.30s are often dropped as fleeting blips.
- Silence-Gating: Fillers detected by the Core ML backend are kept only if they are significantly long (~0.5s) or sit at a pause boundary (within 0.2s of a silence edge). This prevents cutting natural "mid-sentence hmms" that are load-bearing for the sentence structure.
Sources: research/NOTES.md:120-135, apps/desktop/Sources/crisp-filler/main.swift:34-38
Sequence Diagram: Inference Execution
The following diagram shows the interaction between the Swift application and the specialized inference helper.
Conclusion
Filler Word Detection in Crisp is a multi-layered system that balances speed and accuracy. By providing a custom Core ML implementation (Wren) alongside a transcription-based fallback (Whisper), the application ensures that users can achieve high-quality jump-cuts with minimal manual editing, while maintaining the flexibility to optimize for different types of content and performance requirements.