Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
Crisp employs a multi-layered testing strategy designed to ensure the integrity of its native macOS application and its underlying Python-based cleaning engine. The strategy combines automated unit testing, continuous integration (CI) gates, and automated AI-driven code reviews. This approach ensures that core logic—ranging from UI state management to complex audio/video processing—is verified before any code reaches the stable release channel.
The testing architecture is split between the Swift application layer (UI, settings, and subprocess management) and the Python engine layer (audio analysis, cut planning, and FFmpeg graph generation). All changes are first validated in a local development environment using a dedicated "Dev" build channel to avoid disturbing stable installations.
Sources: CLAUDE.md:1-25, CLAUDE.md:65-80
Continuous Integration & Automated Review
Every Pull Request (PR) into the nightly branch undergoes a rigorous automated verification process. This process includes building the Swift application, running the Swift test suite, linting with SwiftLint, and executing the Python engine's unit tests.
Automated Review Agents
Crisp utilizes three automated AI agents to review code changes on every push to an open PR:
- Cubic AI: General code analysis.
- CodeRabbit: AI-driven summary and review findings.
- Scarlet: Specialized code review feedback.
Sources: CLAUDE.md:120-135
Promotion Gate
Stable releases are promoted from the nightly branch to main via an automated script (promote.sh). This promotion is strictly gated by a verification job that must pass all build and test requirements. If the build or tests fail, the promotion to stable is blocked.
Sources: CLAUDE.md:44-55
The diagram above illustrates the path a code change takes from PR creation to reaching the stable branch. Sources: CLAUDE.md:44-60, CLAUDE.md:120-130
Swift Application Testing
The Swift test suite, located in apps/desktop/Tests/CrispTests/, focuses on the application's business logic, state management, and the construction of subprocess commands.
Core Logic Verification
The test suite covers several critical areas:
- Version & Update Logic: Validating numerical version comparisons and build number parsing.
- Settings & Config: Ensuring
EngineConfigandEngineSettingsdecode correctly across versions (forward compatibility) and that default values are preserved. - Cut Preview Math: Calculating removed seconds and keep-segments based on silence thresholds and "breathing room" parameters.
- Resource Management: Testing the
ResourceGovernorto ensure concurrency recommendations respect system memory, CPU core counts, and thermal states.
Sources: CrispTests.swift:1-40, CrispTests.swift:450-510
Resource Governor Test Cases
The ResourceGovernor is tested against various system snapshots to ensure the app doesn't oversubscribe the hardware.
| Scenario | Snapshot Params | Expected Concurrency |
|---|---|---|
| Low Memory | 8GB Total, 0.5GB Available | 1 (Serial) |
| High End (Hardware Encode) | 64GB Total, 32GB Available | 3 (Media Engine Cap) |
| High End (Software Encode) | 64GB Total, 32GB Available | 5 (CPU Core Cap) |
| Thermal Pressure | 64GB Total, 32GB Available, Critical Thermal | 1 (Safe Serial) |
Sources: CrispTests.swift:615-660
Engine Testing (Python)
The cleaning engine's logic is tested using Python's unittest framework. These tests are "stdlib-only," meaning they do not require FFmpeg or Whisper to be installed, allowing them to run quickly in any environment.
Cut Planning and Graph Building
The test_edit.py suite validates the "heart of the engine":
build_keep_segments: Tests the logic that converts detected pauses and fillers into a finalized list of segments to keep. It includes edge cases for overlapping removals and fragments shorter than themin_keepthreshold.- Filter Graph Generation: Pins the exact FFmpeg filter strings generated for audio micro-fades and crossfades.
- Zero-Crossing Snapping: Verifies that interior cut boundaries are correctly nudged to the nearest audio zero-crossing to prevent audible clicks.
Sources: test_edit.py:1-15, test_edit.py:170-220, test_edit.py:270-320
Logging Integrity
The test_enginelog.py suite ensures that the engine's logging system maintains a format compatible with the Swift application's logs, enabling a unified interleaved timeline. It verifies that multi-line tracebacks and tool errors (like FFmpeg failures) are correctly prefixed and recorded.
The class-like structure represents the primary test suites integrated into the CI pipeline. Sources: test_edit.py, test_enginelog.py, CrispTests.swift
ML & Model Dev-Flow Testing
Crisp includes specialized tests for its machine learning integration, specifically for custom filler-detection models (e.g., Wren).
Model Lifecycle Tests
The FillerModelDevFlowTests.swift suite verifies:
- Channel Mapping: Ensuring that Stable builds poll the
mainHugging Face branch, while Nightly/Dev builds poll thenightlystaging branch. - URL Helpers: Validating that URLs for manifests and versioned model files are constructed correctly according to Hugging Face's API.
- Architecture Guards: Ensuring the app only attempts to download and run model types (e.g.,
chunk,sequence) supported by the current build.
Sources: FillerModelDevFlowTests.swift:1-35, FillerModelDevFlowTests.swift:65-80
Manual Verification Workflow
During development, changes are verified against the Crisp Dev build channel.
./dev.sh: Builds and installs "Crisp Dev" side-by-side with the stable version.- Visual Diffs: UI changes are captured via
screencaptureand verified visually. - Real Footage Testing: Features like "Retake Removal" are tested against real-world video footage to identify empirical performance gaps that unit tests might miss (e.g., semantic corrections vs. verbatim repeats).
Sources: CLAUDE.md:65-80, notes.md:60-100
Summary
Crisp's testing strategy is a comprehensive blend of unit-level validation and workflow-integrated verification. By maintaining independent test suites for the Swift UI and the Python processing core, while gating releases behind multi-layered CI checks and AI reviews, the project maintains high stability despite the complexity of native macOS development and audio/video manipulation.