Retake Removal Algorithm
Relevant source files
The following files were used as context for generating this wiki page:
The Retake Removal Algorithm is a core component of the Crisp engine designed to identify and remove redundant video segments caused by "retakes"—instances where a speaker misspeaks and immediately repeats a phrase. By analyzing transcripts and audio silence, the algorithm automatically cuts the "flubbed" first attempt while preserving the corrected version. This feature significantly reduces manual editing effort by targeting the most tedious part of video post-production.
Sources: notes.md:4-7, retake-calibration.md:4-7, apps/desktop/Resources/engine/crisp/config.py:101-104
Core Detection Logic
The algorithm operates primarily on the transcript generated by the Whisper speech-to-text model. It identifies repeated sequences of words (runs) within a specific temporal and linguistic window. To ensure precision and avoid false positives—such as intentional parallel structures or common conversational phrases—the algorithm employs three primary validation signals: Fuzzy Token Matching, Pause-anchoring, and Semantic Gating.
The following diagram illustrates the high-level decision flow for identifying a retake:
1. Fuzzy Token Matching
Natural speech rarely produces identical transcripts across two takes. Whisper may vary contractions ("we're" vs. "were") or word forms ("open" vs. "opens"). The algorithm uses a similarity threshold (RETAKE_TOKEN_SIM = 0.85) to treat these variants as matches. Short function words, however, still require exact matches to prevent genuinely different phrases from being merged.
Sources: notes.md:12-15, apps/desktop/Resources/engine/crisp/config.py:126-133
2. Pause-anchoring and Run Length
The algorithm uses the duration of a repeated word sequence and the presence of silence as its "load-bearing levers."
- Pause-anchoring: A corrected take is validated if it begins immediately after a detected silence (threshold of ~0.3s). This distinguishes a retake from a continuous sentence with a repeat.
- Run Length: Longer verbatim repeats are accepted even without a pause (e.g.,
min_run_no_pause), catching "stumbling restarts" where the speaker corrects themselves without stopping.
Sources: notes.md:16-18, notes.md:57-59, apps/desktop/Resources/engine/crisp/config.py:136-138
3. Semantic Gating
Crisp utilizes the NLContextualEmbedding (via the crisp-embed helper) to calculate on-device sentence embeddings for take comparisons.
- Purpose: It acts as a "rescue" signal for short, pause-less repeats.
- Constraint: Because embeddings can struggle to distinguish redos from parallel lists on short phrases, the semantic score currently only logs data or rescues candidates at a very high bar; it does not veto matches supported by strong word-run evidence.
Sources: notes.md:20-24, retake-research.md:36-41, apps/desktop/Resources/engine/crisp/config.py:112-118
Sensitivity Policies
The algorithm is configured through three distinct sensitivity presets, defined in crisp/config.py. These presets represent full policies governing minimum word counts and pause requirements.
| Preset | min_run | require_pause | min_run_no_pause | sem_min | Description |
|---|---|---|---|---|---|
| Gentle | 5 | True | None | 0.78 | Conservative; requires long matches and silence. |
| Balanced | 4 | True | 7 | 0.78 | Moderate; allows long pause-less repeats. |
| Aggressive | 3 | False | 5 | 0.70 | Catches natural mid-sentence restarts. |
Sources: apps/desktop/Resources/engine/crisp/config.py:119-124, notes.md:46-48
Technical Implementation
The algorithm is implemented in crisp/retake.py and orchestrated within the pipeline.py cleaning loop.
Pipeline Orchestration
The cleaning pipeline follows a sequence of audio analysis and transcript processing:
- Audio Extraction: Extracts WAV for analysis.
- Silence Detection: Identifies potential anchor points (using
RETAKE_ANCHOR_PAUSE). - Transcription: Uses Whisper to generate word-level timestamps.
- Retake Detection:
detect_retakesscans the word list using the selected policy. - Segment Planning:
build_keep_segmentsmerges retake removals with pause and filler removals.
Key Data Structures and Constants
RETAKE_MAX_GAP(2.0s): Maximum time allowed between the flubbed take and the correction.RETAKE_MAX_ABANDON(12 words): Bounds the search by limiting how many words a speaker might "abandon" before starting over.RETAKE_STUTTER(Default: False): Gathers single-word repeats ("the the"); disabled by default to avoid cutting intentional emphasis.
Sources: apps/desktop/Resources/engine/crisp/config.py:134-150
Research and Future Directions
The current algorithm is a "Tier 1" natural-detection system based on verbatim matching and semantic corroboration. Research suggests that "fully-trusted silent auto-cut" of natural retakes is an industry-wide challenge, with competitors often maintaining 66–81% accuracy on unscripted speech.
Future iterations ("Tier 2") consider:
- Discourse Markers: Using markers like "I mean" or "sorry" as additional catch-paths for restarts.
- Verbatim ASR: Moving to fine-tuned models that don't "smooth away" disfluencies like standard Whisper does.
- Disfluency Taggers: Training small Core ML models (e.g., BERT-based) to identify speech repairs directly.
Sources: notes.md:27-30, retake-research.md:11-14, retake-research.md:26-34, retake-research.md:92-98
Summary
The Retake Removal Algorithm balances high recall with aggressive precision filtering to minimize false positives. By combining transcript analysis with acoustic pause-anchoring and semantic similarity, it provides a native on-device solution for cleaning unscripted video. While it currently operates with high trust on clean, scripted repeats, the system is designed to evolve into a multi-signal model capable of handling complex conversational stumbles.
Sources: notes.md:31-35, retake-research.md:104-106