Parsing & Data Decoding

Relevant source files

The following files were used as context for generating this wiki page:

• src/parser.rs
• src/plotter.rs
• src/monitor.rs
• src/export.rs
• src/config.rs

Parsing and Data Decoding in ComChan is a multi-stage process designed to transform raw byte streams from serial ports, BLE services, or RTT/defmt probes into structured telemetry data. The system is built to handle various embedded logging formats, including standard ASCII, key-value pairs, and Zephyr/RTOS-specific log headers, ensuring that numerical data can be extracted for real-time plotting and CSV exportation.

The decoding pipeline operates by first cleaning raw input—removing ANSI escape codes and log metadata—before applying a series of heuristic strategies to identify numerical values. This robust architecture allows ComChan to interpret data ranging from simple bare numbers to complex, multi-channel comma-separated telemetry streams without requiring rigid pre-configuration of the data format.

Data Processing Pipeline

The system follows a linear pipeline to move from raw hardware buffers to the UI and storage layers. Data ingestion handles the reassembly of fragmented lines, while the parser module identifies relevant sensor metrics.

Pipeline Architecture

The diagram below illustrates the flow of data from reception to the final decoding stages.

Decoding Components

The following table summarizes the primary components involved in the decoding process:

Component	Responsibility	Relevant Files
Line Accumulator	Reassembles chunks of bytes into complete strings terminated by newlines.	monitor.rs, plotter.rs
Log Stripper	Removes timestamps, log levels (INFO, ERR), and RTOS-specific headers.	parser.rs
Decoding Heuristics	Applies regex-like logic to find Key:Value, Key=Value, or bare numbers.	parser.rs
Sensor Registry	Maps parsed names to persistent state (color, min/max bounds).	parser.rs, plotter.rs

Sources: src/parser.rs:25-58, src/parser.rs:60-134, src/plotter.rs:163-181

Ingestion & Line Reassembly

Data is received in arbitrary chunks through hardware-specific readers. The monitor.rs and plotter.rs modules maintain internal buffers to accumulate these chunks until a newline character (\n) is encountered.

• Serial Draining: The system utilizes a "drain loop" to pull all available data from the OS buffer before each UI frame, preventing latency in high-speed telemetry streams.
• Buffer Cleaning: The strip_ansi function is used to remove VT100/ANSI escape sequences that embedded devices often use for colored logs, which would otherwise interfere with numerical parsing.

Sources: src/monitor.rs:32-58, src/plotter.rs:420-450

Log Metadata Stripping

Before numerical extraction begins, the system identifies and removes non-telemetry metadata. This is primarily handled by the strip_log_prefixes function.

Prefix Detection Logic

1. Timestamp/Level detection: It looks for patterns like [123.456] <info> module: message.
2. Zephyr Support: Specialized handling for Zephyr RTOS logs (<inf>, <err>, etc.) allows the system to bypass standard log headers to reach the payload.
3. Defmt Integration: The parser can strip defmt module paths and tree separators (e.g., └─, ├─) common in modern Rust embedded logging.

Sources: src/parser.rs:60-134, src/parser.rs:188-200

Multi-Strategy Parsing

The parse_sensor_data function acts as a dispatcher that attempts multiple strategies in order of specificity.

Extraction Methods

• Key-Value Pairs: Handles formats like temp:25.5 or voltage=3.3. It uses rfind to split at the last occurrence of the delimiter to safely handle complex log lines.
• Scientific Notation: The parser specifically allows e and E characters during numeric validation to support large or small floating-point values.
• Keyword Fallback: If no structure is found, the system scans for keywords like temp, gyro, or accel to automatically label bare numbers found in proximity.

Sources: src/parser.rs:203-240, src/parser.rs:296-320

Data Structures & State

Parsed data is encapsulated in the SensorData struct, which maintains the historical and statistical state for each unique telemetry stream.

pub struct SensorData {
    pub name: String,
    pub data: Vec<(f64, f64)>, // (x_timestamp, y_value)
    pub color: Color,
    pub min_value: f64,
    pub max_value: f64,
    pub current_value: f64,
}

Sources: src/parser.rs:25-32

The system automatically assigns colors from a predefined COLORS palette to new sensors based on their insertion order, ensuring distinct visualization in the plotter.

Sources: src/parser.rs:3-23

Exporting & Persistence

Decoded data is routed to two main export paths if configured in MergedConfig:

1. CSV Streaming: The CsvStreamer writes rows containing a timestamp and the values of all currently known sensors. If a sensor value is missing in a specific line, it leaves the column empty.
2. SVG Export: The export_to_svg function takes the HashMap<String, Vec<(f64, f64)>> of decoded points and generates a high-quality visualization, supporting both Light and Dark modes.

Sources: src/export.rs:146-193, src/export.rs:11-30, src/config.rs:232-268

Summary

The Parsing & Data Decoding system provides a flexible bridge between raw embedded communication and high-level data visualization. By combining automated log stripping with multiple heuristic parsing strategies, ComChan can adapt to diverse firmware output formats with minimal user configuration, while maintaining the performance required for real-time telemetry.