Continuous CSV Streaming & Logging

Relevant source files

The following files were used as context for generating this wiki page:

• src/export.rs
• src/config.rs
• src/monitor.rs
• src/plotter.rs
• src/dual_ports.rs
• src/parser.rs

Continuous CSV Streaming and Logging is a core feature of ComChan designed to provide persistent records of serial communication and parsed telemetry data. The system allows users to capture raw serial traffic into log files and simultaneously stream extracted numeric sensor data into structured CSV files for post-processing and analysis.

This system operates across multiple execution modes, including the standard serial monitor, the real-time plotter, and the dual-monitor view. By decoupling raw data capture from parsed data streaming, ComChan ensures that both the original communication context and the structured data insights are preserved.

Sources: README.md, src/monitor.rs:105-117, src/plotter.rs:356-358

System Architecture

The logging and streaming architecture is built on a producer-consumer model where input streams (Serial, BLE, RTT, or Simulated) feed into a central processing loop. This loop branches into two distinct paths: raw byte/string logging and parsed numeric streaming.

Data Flow Overview

The following diagram illustrates how data travels from physical hardware into the various file persistence systems:

The application uses BufWriter for raw logging to minimize I/O overhead and a specialized CsvStreamer struct to manage CSV headers and row alignment.
Sources: src/monitor.rs:430-460, src/plotter.rs:445-470, src/export.rs:136-180

CSV Streaming Logic

The CsvStreamer is responsible for transforming parsed sensor readings into a tabular format. It dynamically handles header generation based on the first set of valid readings received.

Key Components of CsvStreamer

Component	Description	Source
writer	A BufWriter wrapping a file opened in append mode.	src/export.rs:137
headers	A vector of strings identifying the sensor names found in the data stream.	src/export.rs:138
header_written	A boolean flag to ensure the CSV header is only written once at start of stream.	src/export.rs:139
write_row()	Method that aligns incoming (String, f64) pairs with existing headers and writes a row.	src/export.rs:152

Row Alignment Process

When write_row is called, the system performs the following steps:

1. Timestamping: Captures the current local time using get_timestamp().
2. Header Matching: Iterates through the established headers list.
3. Data Filling: If a value for a specific header exists in the current parsed_data, it is written; otherwise, an empty field is inserted to maintain column alignment.
4. Flushing: The internal buffer is flushed to ensure real-time persistence.

Sources: src/export.rs:152-179

Raw Serial Logging

Raw logging captures the literal string output from the communication channel. Unlike CSV streaming, which requires numeric parsing, raw logging preserves ANSI escape codes and non-numeric metadata unless configured otherwise.

Implementation in Monitor and Plotter Modes

In standard monitor mode, logging occurs inside the main read loop. Every chunk of data read from the port is appended to a BufWriter with a local timestamp prefix if verbose mode is enabled.

Dual Monitor Logging

In Dual Monitor mode, the system automatically splits logging into two separate files to prevent data interleaving from different ports. It utilizes the split_filename helper to append _port1 and _port2 suffixes to the user-provided log path.
Sources: src/dual_ports.rs:90-101, src/dual_ports.rs:141-155

Configuration and CLI Options

The logging and streaming behavior is controlled via specific CLI arguments defined in the Args struct.

Option	Argument	Description	Default
Log File	--log <PATH>	Path to save raw serial communication.	None
CSV File	--csv <PATH>	Path to save parsed numeric telemetry.	None
Verbose	--verbose	Includes timestamps and configuration details in log output.	false
Simulate	--simulate	Generates mock data (Temp, Humid, Pressure) for logging/streaming.	false

Sources: src/config.rs:145-147, src/config.rs:177-185, src/config.rs:205

Integration with Data Parser

CSV streaming relies heavily on src/parser.rs to identify key-value pairs in the serial stream. The parser supports multiple formats, which are subsequently structured by the CsvStreamer:

• Comma Separated: Temp: 22.5, Humid: 60.1
• Space Separated KV: ax:1.0 ay:0.5 az:9.8
• Zephyr Logs: Handles stripping of <inf>, <err> tags before streaming to CSV.
• Bare Numbers: Automatically assigns generic labels like "Value" or "Channel X".

Sources: src/parser.rs:146-170, src/parser.rs:188-210

Conclusion

The Continuous CSV Streaming and Logging system in ComChan provides a robust mechanism for both diagnostic preservation and data acquisition. By implementing a dynamic header system in the CsvStreamer and utilizing efficient buffered I/O, the application can handle high-frequency data streams without compromising terminal UI performance. This dual-track approach—logging raw bytes for debugging and streaming parsed data for analysis—makes ComChan a versatile tool for embedded systems development.

Sources: src/export.rs:136-180, src/monitor.rs:470-495