Research Datasets
Relevant source files
The following files were used as context for generating this wiki page:
Research datasets in the Crisp project form the foundational infrastructure for training, evaluating, and benchmarking the Wren and Raven machine learning models. These models are specifically designed for high-performance, on-device filler-word detection and speech repair. By leveraging a mix of primary speech corpora, clinical disfluency datasets, and environmental sound libraries, the project achieves high precision (~0.94) in identifying linguistic "hesitations" like "um" and "uh" at speeds up to 600× real-time.
The research environment is kept strictly separate from the production engine to maintain a standard-library-only footprint in the shipped application. Data processed within this research pipeline is used to generate exported Core ML artifacts that provide an alternative to traditional, resource-heavy Whisper-based transcription for filler detection.
Sources: research/README.md:3-10, research/NOTES.md:7-13, README.md:35-40
Primary Speech Corpora
The project utilizes specialized speech datasets to train models on diverse speaker profiles and varied acoustic environments. These datasets provide the ground truth labels for binary filler-vs-speech classification.
PodcastFillers
PodcastFillers is the primary training source for the Wren model. It contains 145 hours of audio from over 350 speakers. The dataset is structured around human-labeled "uh" and "um" events, providing exact timing within 1-second clips.
Sources: research/NOTES.md:21-31, research/README.md:46-52
SEP-28k
Originally developed by Apple, SEP-28k focuses on stuttering and disfluency events in podcasts. Crisp utilizes this dataset primarily for "hard negatives" to prevent the model from misidentifying non-filler disfluencies as removable words.
Sources: research/NOTES.md:35-42, research/MODEL_RESEARCH.md:92-96
GigaSpeech
GigaSpeech is identified in the roadmap as a crucial dataset for the heavier Raven tier. Its multi-domain nature (podcasts, YouTube, audiobooks) closely matches Crisp’s target domain for real-world talking-head videos.
Sources: research/MODEL_RESEARCH.md:120-125
| Dataset | Total Volume | Use Case | License |
|---|---|---|---|
| PodcastFillers | 145h / 35k events | Primary Training (Wren) | CC |
| SEP-28k | 28k clips | Hard Negatives / Stuttering | CC BY-NC |
| FluencyBank | ~50h | Disfluency variety | Research Use |
| GigaSpeech | 10,000h | Multi-domain ASR (Raven) | Apache 2.0 |
Sources: research/MODEL_RESEARCH.md:88-125
Hard Negative & Synthetic Datasets
To reduce false positives in non-speech environments (such as music or environmental noise), the project incorporates non-vocal datasets to create "hard negatives." This ensures the model does not trigger cuts during transitions or background activity.
Environmental and Music Data
- ESC-50: 2,000 clips across 50 categories (rain, dog barks, etc.) used to ensure the model never fires on environmental sounds.
- MUSDB18 & FMA: Multi-track music datasets used to simulate music-heavy episodes where models frequently over-fire.
Sources: research/MODEL_RESEARCH.md:104-114
Synthetic Negative Generation
The research pipeline employs a technique of mixing speech clips with noise or music at various Signal-to-Noise Ratios (SNR) to generate unlimited training data for edge cases.
The diagram shows the process of generating synthetic negatives by mixing speech with noise/music to train the model to ignore non-filler sounds.
Sources: research/MODEL_RESEARCH.md:145-154
Data Preprocessing & Normalization
Data must be standardized before entering the training pipeline to ensure parity between the Python research environment and the Swift-based inference engine.
Audio Transformation Pipeline
- Resampling: All audio is converted to 16 kHz mono.
- Mel Spectrogram: Transformation using
n_fft=400,hop=160, andn_mels=64. - Log-Mel conversion: Scaling to decibels.
- Standardization: Application of fixed constants (
MEL_MEAN=-18.5658,MEL_STD=17.9252) rather than per-clip statistics to avoid train/serve skew.
Sources: research/NOTES.md:46-52, research/README.md:103-110
Dataset Windowing (Wren v2)
For context-aware models, the SeqWindows dataset serves 4-second windows. Labels are categorized into three buckets based on pause-adjacency:
- Isolated: Silence on both sides (Removable).
- Boundary: Silence on one side.
- Embedded: Buried in speech (Natural - do not cut).
Sources: research/NOTES.md:209-222, research/NOTES.md:231-233
Evaluation and Validation Framework
The project includes custom tools to validate model performance on user-domain footage, ensuring the datasets provide sufficient generalization.
Validation Workflow
The validate module allows for quick "ear-labeling" of new footage to calculate real-world precision and recall.
This diagram illustrates the manual validation loop used to grade model performance against specific user recordings.
Sources: research/README.md:88-101
Performance Benchmarks (Wren v0.0.8)
| Threshold | Precision | Recall | F1 Score |
|---|---|---|---|
| 0.5 | 0.914 | 0.933 | 0.924 |
| 0.7 | 0.944 | 0.894 | 0.918 |
| 0.8 | 0.959 | 0.863 | 0.908 |
Sources: research/filler_classifier/MODEL_CARD.md:29-35
Improvement Roadmap (Tier B)
The future strategy for research datasets focuses on "Data over Architecture." The primary lever for improvement is scaling the negative class coverage through:
- Distillation: Using teacher models (like Whisper or AudioSet taggers) to auto-label large volumes of unlabeled audio.
- Domain Adaptation: Using the
--dataset folderoption to fine-tune models on specific talking-head footage representative of actual app usage.
Sources: research/NOTES.md:280-290, research/README.md:65-68, research/MODEL_RESEARCH.md:195-205
Research datasets are the critical asset enabling Crisp to transition from general-purpose ASR to specialized, efficient audio editing. The combination of human-labeled corpora and synthetic noise generation allows for a model that is both lightweight (~94 KB) and highly precise.