Semantic Gating
Relevant source files
The following files were used as context for generating this wiki page:
Semantic Gating is a secondary validation layer within Crisp's retake detection system designed to distinguish between genuine speech redos (retakes) and intentional parallel structures in a recording. It leverages on-device natural language processing to calculate the semantic similarity between two phrases, acting as a "precision filter" to reduce false positives that traditional verbatim transcript matching might trigger.
This system is particularly critical for handling cases where a user might repeat a phrase for emphasis or as part of a list (parallel structure), which text-only algorithms often misidentify as a fumbled attempt needing a cut. By analyzing the intent behind the words rather than just the characters, the semantic gate helps decide whether a segment should be "rescued" or cut.
Sources: retake-research.md, notes.md:12-25
System Architecture
The Semantic Gating system is implemented as a decoupled helper service that the main Python engine invokes during the video cleaning pipeline.
Core Components
| Component | Technology | Role |
|---|---|---|
| crisp-embed | Swift (NaturalLanguage) | A standalone executable that calculates cosine similarity between phrases using Apple's on-device embeddings. |
| CRISP_EMBED | Environment Variable | Points the Python engine to the bundled crisp-embed binary. |
| NLContextualEmbedding | Apple Framework | Provides 512-dimensional contextual embeddings on macOS 14+. |
| NLEmbedding | Apple Framework | Acts as an offline fallback for older macOS versions (non-contextual). |
Sources: apps/desktop/Package.swift:76-85, retake-research.md:94-110
Integration Flow
The following diagram illustrates how the Python engine interacts with the Swift-based semantic helper to validate a potential retake.
The Python engine calls crisp-embed as a subprocess, passing phrase pairs and receiving a similarity score back as JSON.
Sources: apps/desktop/Sources/crisp-embed/main.swift, notes.md:16-19
Embedding Logic
The crisp-embed tool is responsible for the heavy lifting of semantic analysis. It utilizes Apple's NaturalLanguage framework to generate vector representations of text.
Implementation Details
- Vector Dimension: On macOS 14+, the system uses
NLContextualEmbeddingwhich provides a 512-dimensional vector. - Similarity Metric: The tool calculates the Cosine Similarity between the two vectors.
- Asset Management:
NLContextualEmbeddingmay download an OS-managed asset on first use (similar to the whisper model download). - Execution: The binary is built as a standalone executable to ensure "one system, not two," allowing the engine to leverage native Apple Silicon capabilities without requiring complex Python ML dependencies.
Sources: apps/desktop/Sources/crisp-embed/main.swift, retake-research.md:94-98, apps/desktop/Package.swift:76-85
Operational Heuristics
Empirical findings during development shaped how the semantic gate is used within the detection logic. Research indicated that a "correction" often changes the meaning of a sentence by design (e.g., changing "slow" to "fast"), meaning low similarity scores are not always evidence of a non-redo.
The "Rescue" Policy
Currently, the semantic embedding is used as a corroborating signal rather than a hard veto:
- Never Vetoes: A low similarity score does not automatically disqualify a candidate as a retake.
- Rescues Repeats: It can "rescue" short repeats that lack a clear pause anchor if the semantic similarity is at a deliberately high bar.
- Data Logging: Every candidate's similarity score is logged (
sim=) to gather real-world data for future tuning or potential replacement with a stronger LLM-based "retake judge."
Sources: notes.md:20-25, retake-research.md:104-106
Deployment and Build
The crisp-embed helper is compiled during the standard app build process and bundled into the application's resources.
The build script ensures the helper is signed and placed where the Python engine can find it at runtime.
Sources: apps/desktop/build.sh:34-40, apps/desktop/build.sh:65-68, apps/desktop/build.sh:176-180
Conclusion
Semantic Gating provides a sophisticated layer of intent-awareness to Crisp's editing engine. While the current implementation primarily uses it for data collection and as a limited "rescue" mechanism for pause-less restarts, its architecture as a standalone native helper allows for future scaling—such as moving to full sentence transformers or local LLMs—without re-architecting the core Python processing pipeline.
Sources: notes.md:20-25, retake-research.md:113-119