Model Training Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Model Training Pipeline in Crisp is a dedicated research and development workflow used to create, evaluate, and deploy tiny CNN-based audio classifiers. These models, specifically the "Wren" family, are designed to detect filler words (such as "um" and "uh") with high efficiency and low latency, serving as an on-device alternative to heavy Automatic Speech Recognition (ASR) systems like Whisper for the specific task of filler detection.
The pipeline spans from raw data ingestion and feature extraction to Core ML export and automated hosting on Hugging Face. It is structured into distinct generations: v1 (per-chunk classifier) and v2 (context-aware temporal model). While the training code is dependency-heavy and intended for research environments, it produces optimized, standalone artifacts that are consumed by the Swift-based desktop application.
Sources: research/NOTES.md:1-20, research/README.md:1-15
Pipeline Architecture and Data Flow
The training pipeline follows a linear progression from data preparation to artifact publication. It utilizes 16 kHz mono audio as the primary input and produces binary classification probabilities.
This diagram illustrates the end-to-end flow of the model development lifecycle from raw data to a published artifact.
Sources: research/README.md:18-35, research/NOTES.md:50-100
Data Ingestion and Preparation
The pipeline supports multiple dataset sources for training the classifier, primarily focusing on human-labeled podcast data.
| Dataset | Source | Description | Usage in Pipeline |
|---|---|---|---|
| PodcastFillers | Adobe/Interspeech 2022 | 35k human-labeled "uh"/"um" events; 145h audio. | Primary training source for Wren. |
| SEP-28k | Apple | Stuttering events dataset (labels + scripts only). | Used for "hard negatives" to reduce false positives. |
| Custom Folder | User-provided | Local *.wav + *.fillers.json files. | Domain adaptation for specific footage. |
Sources: research/NOTES.md:23-45, research/README.md:55-80
Feature Extraction
Audio is converted into a normalized log-mel representation before being fed into the model. This process is shared between the training scripts and the production inference helper (crisp-filler).
- Sample Rate: 16 kHz mono.
- Transform: MelSpectrogram (n_fft=400, hop=160, n_mels=64).
- Normalization: Fixed-constant normalization using
MEL_MEAN(-18.5658) andMEL_STD(17.9252). - Context Windows: A "chunk" consists of 25 mel frames representing 0.25 seconds of audio.
Sources: research/NOTES.md:52-60, apps/desktop/Sources/crisp-filler/main.swift:18-30
Training and Model Generations
Generation v1: Per-Chunk Classifier
The v1 model is a small 2-D CNN (~75k parameters) that performs binary classification (filler vs. speech) on individual 0.25s audio chunks. It uses BCEWithLogitsLoss with a pos_weight to handle class imbalance.
Sources: research/NOTES.md:61-68, research/README.md:30-33
Generation v2: Context-Aware Temporal Model (WrenSeq)
The v2 pipeline introduces a dilated Temporal Convolutional Network (TCN) with ~129k parameters. Unlike v1, v2 processes a sequence of log-mel frames (4s windows during training) to learn temporal context, which helps distinguish between "removable" fillers and "natural" mid-sentence hesitations.
Sources: research/NOTES.md:190-210, research/README.md:34-36
Evaluation and Validation
Evaluation is conducted on a held-out, episode-disjoint test split. The pipeline emphasizes Precision over Recall to avoid "over-cutting" real speech.
The evaluation flow ensures that models are not only statistically accurate on benchmarks but also performant on real-world footage.
Sources: research/NOTES.md:69-75, research/filler_classifier/v1/validate.py:44-70
Key Validation Metrics (Wren v0.0.8)
| Threshold | Precision | Recall | F1 Score |
|---|---|---|---|
| 0.5 | 0.914 | 0.933 | 0.924 |
| 0.7 (Default) | 0.944 | 0.894 | 0.918 |
| 0.8 | 0.959 | 0.863 | 0.908 |
Sources: research/filler_classifier/MODEL_CARD.md:27-33
Export and Publication
Core ML Export
Models are exported to a single-file .mlmodel artifact. This process typically requires a specialized environment (e.g., Python 3.10) because specific tools like coremltools may have version restrictions.
- Input Shape:
[1, 1, 64, 25]for chunk models. - Parity Testing: The export script verifies that the PyTorch and Core ML outputs match within a tolerance of ~1e-9.
Sources: research/README.md:95-105, research/NOTES.md:76-80
Automated Distribution via Hugging Face
The publish_hf.py script manages the versioning and distribution of models.
- Versioning: Version numbers follow the format
0.0.N, whereNis the repository's commit count. - Channels: Models land on a
nightlybranch for staging and are promoted tomain(Stable) only after testing. - Metadata: Every release includes the
.mlmodel, raw PyTorch weights (.pt), and aconfig.jsoncontaining recommended threshold and framing parameters.
Sources: research/filler_classifier/publish_hf.py:34-60, research/NOTES.md:144-155
Model Research and Improvement Roadmap
Recent research focuses on "Data over Architecture" to address common failure modes like music over-firing.
| Technique | Status | Impact | Description |
|---|---|---|---|
| Hard Negatives | Implementation in v3 | High | Using SEP-28k music/noise clips as negatives. |
| SpecAugment | P0 Priority | High | Time and frequency masking for speech robustness. |
| Focal Loss | Proposed | Medium | Focuses training on "hard" examples (boundary fillers). |
| Whisper Encoder | Proposed | High | Using Whisper's acoustic intelligence for features. |
Sources: research/MODEL_RESEARCH.md:107-160, research/NOTES.md:252-270
Summary
The Model Training Pipeline provides a robust framework for developing specialized, ultra-lightweight audio classifiers. By combining rigorous benchmark evaluation with real-world validation and an automated deployment flow, the pipeline ensures that the Crisp app maintains high-quality filler detection while remaining efficient enough for on-device execution on Apple Silicon.