History & Analytics

Relevant source files

The following files were used as context for generating this wiki page:

• analytics-dashboard.md
• CLAUDE.md
• apps/desktop/Resources/engine/NOTES.md
• research/NOTES.md
• README.md

The History and Analytics system in Crisp provides a comprehensive record of video cleaning operations and their associated performance metrics. It serves as a bridge between the core processing engine and the user interface, allowing users to track their editing time saved and review past processing logs. The system is designed with a privacy-first approach, keeping all data on-device.

Sources: analytics-dashboard.md:1-12, CLAUDE.md:144-150

Data Persistence & History Store

The primary data layer for tracking user activity is the HistoryStore. This component maintains a persistent record of every video cleaning session conducted through the application, whether initiated via the main app, the watch folder, or macOS Shortcuts.

Storage Architecture

• Format: The system currently utilizes a JSONL (JSON Lines) file located at ~/.crisp*/history.jsonl.
• Future Migration: There is a planned transition to a local SQLite database if the history file size grows significantly or if more complex querying/aggregation becomes necessary for advanced analytics.
• Privacy: In alignment with the project's philosophy, all history and log data is stored strictly on the user's local machine.

Sources: analytics-dashboard.md:21-28, CLAUDE.md:144-150

History Entry Schema

Each record in the history represents a single HistoryEntry. The following table details the fields typically captured during a "clean" operation:

Field	Description
Timestamp	The date and time the cleaning operation occurred.
Input Name	The name of the original source video file.
Original Seconds	Total duration of the input video before processing.
New Seconds	Total duration of the resulting "cleaned" video.
Saved Seconds	The difference between original and new duration (Time Saved).
Fillers Removed	Count of filler words (um, uh, etc.) detected and cut.
Pauses Removed	Count of silence/pause segments removed.
Retakes Removed	Count of repeated takes identified and pruned.
Sensitivity	The cutting knobs/strength settings used for the operation.
Channel	The app channel (Stable, Nightly, or Dev) that performed the work.

Sources: analytics-dashboard.md:21-25, apps/desktop/Resources/engine/NOTES.md:46-52

Analytics Dashboard

The Analytics Dashboard is a visualized representation of the data stored in the HistoryStore. It is intended to prove the application's value to the user by aggregating metrics over time.

Key Visualizations

• Time Saved Headline: A prominent display of cumulative time saved (e.g., "You've saved 4h 12m").
• Trends over Time: Bar charts displaying counts of retakes, pauses, and fillers removed per week or month.
• Efficiency Metrics: Summary of average percentage shorter and the "biggest single save" achieved.

Sources: analytics-dashboard.md:6-20

The following diagram illustrates the flow of data from a cleaning operation into the History and Analytics system:

System Logging

Crisp maintains a unified logging system that bridges the Swift desktop application and the Python processing engine.

Log File Management

• Location: Files are stored at ~/.crisp*/logs/<yyyy-MM-dd>.log.
• Rotation: CrispApp automatically prunes log files older than 30 days.
• Concurrency: The system uses a serial-queue and O_APPEND to allow the main app, watchers, and parallel engine processes to log safely to the same file.

Sources: CLAUDE.md:155-161

Integrated Timeline

The Python engine (via crisp/enginelog.py) and the Swift app share the same line format to create a single merged timeline. This is crucial for debugging the pipeline:

1. Swift Logs: App lifecycle events, UI interactions, and subprocess spawning.
2. Python Logs: ffmpeg commands, whisper exit codes, and raw stderr on failures.
3. Tracebacks: Unexpected Python exceptions are captured as NDJSON error objects and logged as tracebacks.

Sources: CLAUDE.md:162-170

Performance & Research Metrics

For experimental features like the "Wren" filler-detection model, the system tracks specific performance diagnostics.

Model Diagnostics

When utilizing custom ML models, the engine logs:

• Inference Speed: Real-time multipliers (e.g., Wren at ~600x real-time).
• Accuracy Metrics: Thresholds, frame counts, and span detection counts.
• Execution Metadata: Model version, generation (e.g., v2 sequence vs v1 chunk), and backend type (whisper vs coreml).

Sources: research/NOTES.md:7-14, research/NOTES.md:244-248

Summary

History and Analytics in Crisp provide transparency into the app's performance and value. By persisting processing metrics in a local history.jsonl file and unifying logs across Swift and Python layers, the system enables features like the Analytics Dashboard and provides developers with the diagnostic data needed to refine ML models like Wren. All data remains strictly local to the user's Mac, maintaining the project's commitment to privacy.

Sources: analytics-dashboard.md:6-12, CLAUDE.md:144-150