Video Processing Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
- apps/desktop/Resources/engine/clean_video.py
- apps/desktop/Resources/engine/crisp/pipeline.py
- apps/desktop/Resources/engine/crisp/encode.py
- apps/desktop/Resources/engine/crisp/config.py
- apps/desktop/Resources/engine/crisp/timeline.py
- apps/desktop/Sources/CrispCore/Engine/CleanRunner.swift
- apps/desktop/Resources/engine/NOTES.md
The Video Processing Pipeline is the core engine of Crisp, responsible for transforming raw screen-recordings and talking-head videos into "crisp" versions by removing silences, filler words, and repeated takes. The pipeline is implemented primarily in Python as a standalone engine that shells out to high-performance tools like ffmpeg and whisper.cpp. It is designed to be non-destructive, always backing up original files before processing and outputting a new cleaned file or an editor-ready project.
Sources: CLAUDE.md, apps/desktop/Resources/engine/clean_video.py:1-18, apps/desktop/Resources/engine/crisp/pipeline.py:1-4
High-Level Architecture
The pipeline follows a layered architecture where a Swift-based macOS application drives a Python engine as a subprocess. The Python engine exposes a CLI via clean_video.py, which communicates back to the Swift app using NDJSON (Newline Delimited JSON) events on stdout.
Execution Flow
The sequence diagram below illustrates the interaction between the Swift UI layer, the Python CLI wrapper, and the core processing pipeline.
The pipeline orchestrates several specialized modules to handle detection, editing, and encoding.
Sources: CLAUDE.md, apps/desktop/Sources/CrispCore/Engine/CleanRunner.swift:115-135, apps/desktop/Resources/engine/crisp/pipeline.py:350-550
Pipeline Stages
The processing is divided into several discrete stages, with progress reported to the user at each step.
1. Pre-processing and Analysis
Before any cutting occurs, the system verifies the source file and prepares it for analysis.
- Backup: Optionally copies the original file to an
_originalsfolder or a user-specified directory. - FPS Normalization: Detects Variable Frame Rate (VFR) sources—common in screen recordings—and determines if normalization to a Constant Frame Rate (CFR) is required to maintain A/V sync.
- Audio Extraction: Extracts a 16kHz mono WAV file from the source for silence detection and transcription.
Sources: apps/desktop/Resources/engine/crisp/pipeline.py:397-434, apps/desktop/Resources/engine/crisp/config.py:73-77
2. Detection (Pauses, Fillers, and Retakes)
The detection stage identifies what segments of the video should be removed based on user settings.
| Detection Type | Tool/Method | Description |
|---|---|---|
| Pauses | ffmpeg silencedetect | Finds silence based on audio energy below a noise threshold (default -30dB). |
| Filler Words | whisper.cpp or Core ML | Identifies "um", "uh", "hmm" etc. using either a full speech-to-text model or a fast on-device classifier. |
| Retakes | Transcript Matching | Detects repeated runs of words (stutters or flubbed takes) and keeps only the corrected version. |
Sources: apps/desktop/Resources/engine/crisp/pipeline.py:465-514, apps/desktop/Resources/engine/crisp/config.py:11-53
3. Planning and Smoothing
Once detection is complete, the engine builds a list of "keep segments."
- Zero-Crossing Snap: Interior cut boundaries are nudged to the nearest audio zero-crossing (within a $\pm$12ms window) to prevent audible clicks at splice points.
- Silence Gating: Fillers detected via the fast Core ML backend are gated; they are only cut if they are long ($\ge$0.5s) or adjacent to a silence boundary.
Sources: apps/desktop/Resources/engine/crisp/pipeline.py:532-540, apps/desktop/Resources/engine/NOTES.md:27-38, apps/desktop/Resources/engine/crisp/config.py:116-118
4. Rendering and Export
The final stage produces the output, either as a rendered video or an editor project.
Rendered Video
The engine uses ffmpeg to trim and concatenate the kept segments. It attempts hardware-accelerated HEVC encoding (Apple VideoToolbox) by default, falling back to software encoding if necessary to preserve 10-bit color depth or HDR10 metadata.
Editor Project (FCPXML)
Alternatively, Crisp can export an .fcpxml timeline for professional editors like DaVinci Resolve. This avoids re-encoding the main footage and allows users to fine-tune every cut manually.
Sources: apps/desktop/Resources/engine/crisp/pipeline.py:560-610, apps/desktop/Resources/engine/crisp/timeline.py:8-25
Configuration Parameters
The pipeline is highly configurable through CLI flags and configuration files.
| Parameter | Default | Description |
|---|---|---|
--pause | 0.6s | Cut silences longer than this duration. |
--noise | -30dB | Loudness threshold for silence detection. |
--fade-ms | 10ms | Audio micro-fade at cuts to prevent clicks. |
--video-codec | hevc | Encoder choice (h264, hevc, vp9). |
--hardware | True | Use Apple VideoToolbox hardware acceleration. |
--color-depth | auto | Matches source (8-bit or 10-bit). |
Sources: apps/desktop/Resources/engine/clean_video.py:64-110, apps/desktop/Resources/engine/crisp/config.py:55-87
Technical Implementation Details
Frame-Accurate Trimming
To ensure cuts are frame-accurate and A/V stays in sync, the pipeline uses integer frame indices for all internal calculations. In FCPXML exports, time values are represented as exact integer-frame rational fractions (e.g., frames * den / num).
Sources: apps/desktop/Resources/engine/crisp/timeline.py:15-25, apps/desktop/Resources/engine/crisp/pipeline.py:277-290
Audio Smoothing Mechanisms
The pipeline implements three distinct phases of audio smoothing to eliminate "janky" jump-cuts:
- Audio Micro-fade: A short
afade(default 10ms) is applied to the beginning and end of every kept segment. - Matched A/V Crossfade: Optionally dissolves segments using
xfadeandacrossfadeover the same duration. - Zero-crossing Snap: Nudges cut boundaries to points of zero amplitude.
Sources: apps/desktop/Resources/engine/NOTES.md:12-38, apps/desktop/Resources/engine/crisp/config.py:64-67
Summary
The Video Processing Pipeline provides a robust, automated workflow for cleaning vocal recordings. By combining high-level Swift orchestration with low-level Python and C++ (Whisper/FFmpeg) tools, it achieves high performance while maintaining professional standards like HDR preservation and non-destructive editor integration. The system's modularity allows for diverse detection backends (Whisper vs. Core ML) and various output formats (Rendered MP4 vs. FCPXML).