Model Evaluation & Export
Relevant source files
The following files were used as context for generating this wiki page:
Model Evaluation and Export constitutes the bridge between the high-dependency research environment and the lightweight, production-ready macOS application. This system ensures that custom ML models—primarily for filler word detection—are rigorously tested for precision and recall before being converted into Apple's Core ML format for on-device execution.
The process involves training on datasets like PodcastFillers, evaluating on held-out test splits, and validating performance against real-world footage. Once a model meets quality standards, it is exported as an .mlmodel file and published to a model repository (Hugging Face) alongside metadata that describes its operational parameters.
Sources: research/README.md:1-15, research/NOTES.md:14-25
Evaluation Framework
The evaluation framework consists of multiple layers of testing, from automated metric calculations on research datasets to graphical performance dashboards and manual ground-truth validation.
Performance Metrics & Dashboard
The primary metrics used to judge model quality are Precision, Recall, and F1-score. A native macOS dashboard named FillerBench allows developers to visualize these metrics and perform threshold sweeps.
- Precision: Measures how many of the model's predicted cuts were actual fillers.
- Recall: Measures how many of the real fillers in the audio were successfully caught by the model.
- F1-Score: The harmonic mean of precision and recall, used as the primary selection criterion for best-performing checkpoints.
Sources: research/filler_classifier/v1/validate.py:10-18, research/FillerBench/README.md:1-15, research/NOTES.md:68-75
The diagram above illustrates the flow of data through the evaluation pipeline to produce final performance metrics. Sources: research/FillerBench/README.md, research/filler_classifier/v1/validate.py
Ground-Truth Validation (validate.py)
To ensure the model generalizes to the user's specific domain (e.g., talking-head footage), a validation script is used to grade predictions against manual labels.
- Prepare: A specific window of audio is extracted (e.g., 2-3 minutes).
- Label: The developer manually identifies every "um" and "uh" in a
labels.jsonfile. - Score: The model's predictions are compared against the manual labels to identify False Positives (over-cutting) and False Negatives (misses).
Sources: research/filler_classifier/v1/validate.py:23-45, research/README.md:105-125
Core ML Export Pipeline
The export process transforms PyTorch models into optimized .mlmodel files suitable for the Apple Neural Engine. Because coremltools has specific version requirements, this process often runs in a dedicated environment.
Model Architectures
The project supports two primary generations of filler models:
| Generation | Architecture | Input Shape | Purpose |
|---|---|---|---|
| v1 (Wren v1) | 2D CNN (Chunk) | [1, 1, 64, 25] | Analyzes 250ms chunks in isolation. |
| v2 (Wren v2) | Dilated TCN (Sequence) | [1, 64, T] | Context-aware temporal analysis of full sequences. |
Sources: research/NOTES.md:16-25, research/NOTES.md:200-215, apps/desktop/Sources/crisp-filler/main.swift:23-45
Export Metadata (config.json)
Every exported model is accompanied by a config.json sidecar. This ensures the production app knows how to process audio to match the model's training features without hardcoding values in the Swift source code.
- Normalization: Fixed constants
MEL_MEANandMEL_STD. - Framing:
N_FFT,HOP_LENGTH, andN_MELS. - Thresholds:
recommended_thresholdandmin_fillerduration.
Sources: research/filler_classifier/publish_hf.py:59-80, apps/desktop/Sources/crisp-filler/main.swift:18-35
The sequence diagram shows the transition from a research model to a production-ready Core ML artifact. Sources: research/README.md:90-103, research/filler_classifier/publish_hf.py:110-125
Publishing and Distribution
Models are distributed via Hugging Face using a channel-based system that mirrors the main application's release flow (Stable, Nightly, and Dev).
Release Channels
- Nightly Branch: Used for staging freshly trained models for testing.
- Main Branch (Stable): Contains promoted models that have been verified in Nightly.
- Promotion: The
promote_model.pyscript performs a tree-copy fromnightlytomainto avoid 3-way merge conflicts.
Sources: research/filler_classifier/publish_hf.py:20-35, research/NOTES.md:145-160
Versioning Scheme
Model versions follow a 0.0.N scheme where N is the commit count of the repository. This ensures a monotonic, deterministic versioning history.
Sources: research/filler_classifier/publish_hf.py:46-56, research/NOTES.md:148
Conclusion
The Model Evaluation & Export system provides the necessary rigor to ensure that the "Wren" family of models provides high-precision filler detection on-device. By separating research dependencies from production code and using standardized metadata configurations, the project allows for rapid iteration of ML models without requiring constant updates to the core Swift application.