Log Pipeline & Stream Services
Relevant source files
The following files were used as context for generating this wiki page:
The Log Pipeline & Stream Services form the backbone of real-time data ingestion and enrichment in Keynobi. This system is responsible for capturing raw log streams—primarily from Android's adb logcat—and transforming them into structured, actionable data for the user interface. The service handles high-frequency events through a multi-stage Rust pipeline that performs package resolution, crash grouping, and JSON extraction before batching entries for the frontend.
Key responsibilities include maintaining a 50,000-entry ring buffer in the backend to ensure memory stability, performing zero-allocation filtering on the hot path, and managing process lifecycle events through synthetic separator entries. This architecture ensures that the application remains responsive even during heavy log bursts while providing developers with deep insights into application behavior, such as automatic crash detection and network request categorization.
Architectural Overview
The logging architecture is split between a high-performance Rust backend and a reactive SolidJS frontend. The backend manages the heavy lifting of process spawning, line parsing, and data enrichment, while the frontend handles virtualization and user-driven filtering.
Data Flow
The following diagram illustrates the flow of a log line from the Android device through the enrichment pipeline to the UI.
Backend Log Pipeline
The LogPipeline is the central processing unit for log enrichment. It runs a series of processors on every incoming RawLogLine to produce a ProcessedEntry.
Pipeline Components
The pipeline consists of four standard processors executed in a specific order:
| Processor | Responsibility | Key Logic |
|---|---|---|
| PackageResolver | Maps PIDs to Packages | Uses a PID-to-package map built from ActivityManager events and adb shell ps. |
| CrashAnalyzer | Detects & Groups Crashes | Identifies FATAL EXCEPTION, ANRs, and native signals; groups stack trace lines. |
| JsonExtractor | Payload Detection | Uses a cheap heuristic to find { and " before attempting to parse JSON bodies. |
| CategoryClassifier | Categorization | Assigns categories like Network, Lifecycle, or Database based on log tags. |
Sources: src-tauri/src/services/log_pipeline.rs:180-305
Crash and Lifecycle Analysis
The CrashAnalyzer identifies critical failure states. It uses heuristics to determine when a crash group starts and ends, ensuring that entire stack traces are linked by a crash_group_id. Simultaneously, the pipeline generates lifecycle separators when it detects process starts or deaths via ActivityManager tags.
Stream Management and Performance
To handle high-frequency log output without overwhelming the UI or the IPC bridge, the system employs several optimization strategies.
Ring Buffer and Batching
The LogStore maintains a ring buffer capped at 50,000 entries. This ensures that memory usage remains flat regardless of the session duration. For IPC communication, the backend batches entries into groups of 500 (MAX_BATCH_SIZE) and emits them every 100ms.
Hot Path Filtering
The backend implements a LogcatFilter that performs zero-allocation, case-insensitive substring searches. By filtering on the Rust side before emitting to the frontend, the application significantly reduces IPC overhead and frontend rendering load.
#[inline]
fn ci_contains(haystack: &str, needle: &str) -> bool {
if needle.is_empty() { return true; }
if haystack.len() < needle.len() { return false; }
let nb = needle.as_bytes();
haystack.as_bytes().windows(nb.len()).any(|window| {
window.iter().zip(nb).all(|(&h, &n)| h.to_ascii_lowercase() == n)
})
}
Sources: src-tauri/src/services/logcat.rs:100-112, src-tauri/src/services/logcat.rs:185-195
Frontend Integration
The frontend consumes processed log entries through Tauri events and manages user interactions via specialized components and stores.
LogViewer Component
The LogViewer provides a virtualized interface for viewing both real-time logcat streams and historical build logs. It includes features for:
- Level Filtering: Filtering by
Error,Warn,Info, etc. - Source Filtering: Filtering by the specific log source (e.g.,
gradle,lsp). - Search: Case-insensitive message search.
- Auto-scroll: Toggleable logic that pauses when the user scrolls up to inspect previous entries.
Build Log Management
Build logs are managed similarly but with specific metadata. The build.store.ts handles the batching of build lines and converts them into LogEntry objects for the viewer.
| Feature | Implementation |
|---|---|
| Max Entries | Capped at 10,000 lines for build logs. |
| Batch Flush | Flushed every 50ms to prevent UI stuttering. |
| Formatting | Prepends file, line, and column information if available. |
Sources: src/components/common/LogViewer.tsx:145-215, src/stores/build.store.ts:85-115
Conclusion
The Log Pipeline & Stream Services provide a robust, high-performance solution for Android development telemetry. By leveraging Rust's efficiency for the data-intensive enrichment pipeline and SolidJS's reactivity for the presentation layer, Keynobi offers a seamless experience for real-time log monitoring and post-build analysis. The system's ability to automatically detect crashes and categorize network traffic significantly reduces the manual effort required for debugging complex Android applications.