Relevant source files

The following files were used as context for generating this wiki page: - [src/output/mod.rs](https://github.com/oritwoen/vuke/blob/main/src/output/mod.rs) - [src/output/console.rs](https://github.com/oritwoen/vuke/blob/main/src/output/console.rs) - [src/output/query_format.rs](https://github.com/oritwoen/vuke/blob/main/src/output/query_format.rs) - [src/output/storage.rs](https://github.com/oritwoen/vuke/blob/main/src/output/storage.rs) - [src/output/multi.rs](https://github.com/oritwoen/vuke/blob/main/src/output/multi.rs) - [src/main.rs](https://github.com/oritwoen/vuke/blob/main/src/main.rs) - [AGENTS.md](https://github.com/oritwoen/vuke/blob/main/AGENTS.md)

# Output Formats The `vuke` output system is responsible for handling the results of key generation and analysis. It provides a flexible architecture for presenting data in various formats, catering to different use cases from immediate user feedback on the command line to large-scale data persistence for offline analysis. The system is designed around a central `Output` trait, which defines a common interface for all output handlers. This modularity allows `vuke` to support multiple output destinations simultaneously, such as printing to the console while also writing to a structured Parquet file. The primary output formats include human-readable console text, machine-readable formats like JSON and CSV for query results, and a high-performance Parquet-based storage system for handling terabyte-scale datasets. ## Core Abstraction: The `Output` Trait The foundation of the output system is the `Output` trait, which standardizes how generated keys and matched hits are processed. Any struct implementing this trait can function as an output handler within the application. *Sources: [src/output/mod.rs:18-34]()* ### Trait Definition ```rust // Source: src/output/mod.rs:21-34 pub trait Output: Send + Sync { /// Output a key (no matcher, output all keys). fn key(&self, source: &str, transform: &str, derived: &DerivedKey) -> Result<()>; /// Output a match hit (matcher found target). fn hit( &self, source: &str, transform: &str, derived: &DerivedKey, match_info: &MatchInfo, ) -> Result<()>; /// Flush any buffered output. fn flush(&self) -> Result<()>; } ``` ### Methods | Method | Description | | :------ | :------------------------------------------------------------------------------------------------------ | | `key()` | Called when generating all keys (e.g., `vuke generate`). It receives the source input, the transform used, and the full derived key details. | | `hit()` | Called when a generated key's address matches a target in the provided list (e.g., `vuke scan`). It receives the same data as `key()` plus information about the match. | | `flush()` | Ensures that any buffered data is written to its destination before the program exits. | *Sources: [src/output/mod.rs:23-34]()* ### Implementations The following diagram shows the `Output` trait and its primary implementations within the `vuke` codebase. ```mermaid classDiagram direction LR class Output { <> +key(source, transform, derived) +hit(source, transform, derived, match_info) +flush() } class ConsoleOutput { -writer: Mutex> -verbose: bool } class StorageOutput { -inner: Arc } class MultiOutput { -outputs: Vec> } ConsoleOutput --|> Output StorageOutput --|> Output MultiOutput --|> Output MultiOutput o-- "0..*" Output : contains ``` This diagram illustrates the main components of the output system: the central `Output` trait and its concrete implementations for console, storage, and composite outputs. *Sources: [src/output/console.rs:11](), [src/output/storage.rs:30](), [src/output/multi.rs:7]()* ## Output Handlers ### ConsoleOutput `ConsoleOutput` is the default handler, printing results directly to the standard output or a specified file. It supports two modes: a compact, single-line format and a verbose, multi-line format. *Sources: [src/output/console.rs]()* | Mode | Trigger | Format Description - | :-------- | :------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Compact** | Default | A CSV-style line: `source,transform,privkey,address_compressed`. This is useful for quick parsing or piping to other tools. - | | **Verbose** | `--verbose` flag | A YAML-like, multi-line format that displays all derived key information, including all address types, WIF formats, and the private key. This is ideal for detailed inspection of a single key. | *Sources: [src/output/console.rs:65-82](), [src/main.rs:114]()* When a match is found (`hit`), the output is always verbose and decorated with a "HIT" banner to make it highly visible. *Sources: [src/output/console.rs:88-106]()* ### StorageOutput Enabled with the `--features storage` compile-time flag, `StorageOutput` provides a robust solution for persisting massive volumes of key data to disk using the Apache Parquet format. This is essential for large-scale research and analysis. *Sources: [src/output/storage.rs](), [src/main.rs:119](), [AGENTS.md]()* #### Key Features - **Scalability**: Designed to handle terabytes of data by writing records in batches to Parquet files. - **Structured Data**: Saves results in a structured schema, making it easy to query later with SQL-based tools. - **Automatic Chunking**: Rotates output files based on either the number of records or the file size to keep individual files manageable. - **Compression**: Supports various compression algorithms to balance write speed and disk usage. - **Hive Partitioning**: Organizes files into a directory structure based on the transform used and the date, which optimizes query performance. *Sources: [src/output/storage.rs:37-44, 71-143](), [README.md]()* The following diagram shows the process of writing a record to Parquet storage. ```mermaid flowchart TD A[Source/Transform generates key] --> B{StorageOutput::key()}; B --> C[Create ResultRecord]; C --> D[Acquire Mutex on ParquetBackend]; D --> E[backend.write_batch()]; E --> F[Release Mutex]; F --> G[Increment record counter]; subgraph ResultRecord direction LR C1[source] C2[transform] C3[timestamp] C4[private_key] C5[public_keys] C6[addresses] end C --> C1 & C2 & C3 & C4 & C5 & C6 ``` This flow shows how a generated key is packaged into a `ResultRecord` and written to a Parquet file via the `ParquetBackend`. *Sources: [src/output/storage.rs:194-245]()* #### Configuration The behavior of `StorageOutput` is controlled via CLI arguments when the `generate` or `scan` commands are used. | CLI Argument | Description | Default Value | | --------------------- | --------------------------------------------------- | ------------- | | `--storage ` | Enables Parquet output to the specified directory. | - | | `--chunk-records` | Max records per Parquet file before rotating. | `1000000` | | `--chunk-bytes` | Max file size (e.g., `100M`, `1G`) before rotating. | `100M` | | `--compression` | Compression algorithm to use. | `zstd` | | `--compression-level` | Zstd compression level (1-22). | `3` | *Sources: [src/main.rs:119-142]()* ### MultiOutput The `MultiOutput` struct is a composite output handler that wraps multiple `Output` implementations. When its methods (`key`, `hit`, `flush`) are called, it iterates through all contained handlers and calls the corresponding method on each one. This allows the application to send results to multiple destinations simultaneously, for example, printing to the console while also saving to a Parquet file. This pattern is used in `main.rs` when both a console output (default or via `--output`) and a storage output (`--storage`) are configured. *Sources: [src/output/multi.rs:7-11](), [src/main.rs:509-512]()* The diagram below illustrates how `MultiOutput` combines `ConsoleOutput` and `StorageOutput`. ```mermaid sequenceDiagram participant Main as "main()" participant Multi as "MultiOutput" participant Console as "ConsoleOutput" participant Storage as "StorageOutput" Main->>Multi: key(data) activate Multi Multi->>Console: key(data) Multi->>Storage: key(data) deactivate Multi ``` This sequence shows `MultiOutput` delegating a `key` call to both of its child outputs. *Sources: [src/output/multi.rs:13-18]()* ### Query Result Formatting When using the `vuke query` command (enabled with the `storage-query` feature), the results from the SQL query against the Parquet store can be formatted into several standard formats. This functionality is handled by the `output::query_format` module. *Sources: [src/output/query_format.rs]()* The available formats are defined by the `OutputFormat` enum: ```rust // Source: src/output/query_format.rs:9-14 #[derive(Clone, Copy, Debug, Default, PartialEq)] pub enum OutputFormat { #[default] Table, Json, Csv, } ``` | Format | CLI Flag | Description - | | :------ | :------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- | | `table` | `--format table` (default) | A human-readable, UTF-8 formatted table with colored output for different data types. | | `json` | `--format json` | A JSON array of objects, where each object represents a row. | | `csv` | `--format csv` | RFC 4180 compliant CSV output, with proper escaping for special characters. | *Sources: [src/output/query_format.rs:17-25, 33-90, 126-170, 173-221](), [src/main.rs:388-390]()* ## Conclusion The output system in `vuke` is a well-architected component that provides significant flexibility for handling generated data. By abstracting output logic behind the `Output` trait, the system easily accommodates different user needs, from simple, real-time feedback via `ConsoleOutput` to scalable, long-term data warehousing with `StorageOutput`. The inclusion of `MultiOutput` demonstrates a thoughtful design that allows these different strategies to be composed, while the dedicated `query_format` module ensures that stored data can be easily extracted and integrated with other tools and workflows. This makes the output system a critical and powerful part of the `vuke` research toolkit.