Noumenon Gleaner (Rust CLI)
Relevant source files
The following files were used as context for generating this wiki page:
Noumenon Gleaner is a data extraction tool written in Rust, designed for the Noumenon project to extract data schemas and files. It serves as a specialized CLI utility that processes structured CSV data to generate TypeScript interfaces, ensuring type safety between data sources and frontend applications.
Sources: packages/noumenon-gleaner/README.md, packages/noumenon-gleaner/package.json:4-6, AGENTS.md:27-28
System Architecture
The Gleaner is structured as a Rust-based CLI tool within a monorepo. It utilizes a modular design consisting of a CLI argument parser, a schema builder, and a TypeScript generator. The tool is integrated into the broader project pipeline via noumenon-librarian, which acts as an orchestrator to run the gleaner against raw data.
Sources: packages/noumenon-gleaner/src/main.rs:1-4, packages/noumenon-librarian/README.md:11-12
Execution Flow
The following diagram illustrates the high-level execution flow when the CLI is invoked:
The application flow begins with argument parsing via clap and proceeds to build schemas from input CSV files before generating TypeScript representations.
Sources: packages/noumenon-gleaner/src/main.rs:8-25, packages/noumenon-gleaner/Cargo.toml:7
Core Components
1. Schema Builder
The SchemaBuilder is responsible for analyzing input CSV files and constructing an internal representation of the data schema. It validates the file structure and tracks referenced files to ensure data integrity.
Sources: packages/noumenon-gleaner/src/main.rs:35-36
2. TypeScript Generator
The TypeScriptGenerator takes the schemas produced by the builder and converts them into TypeScript interfaces. This enables the frontend to consume the extracted data with strong typing.
Sources: packages/noumenon-gleaner/src/main.rs:39-44
3. CLI Interface
Built using the clap crate with the derive feature, the CLI handles user input for file paths and configuration.
Sources: packages/noumenon-gleaner/Cargo.toml:7, packages/noumenon-gleaner/src/main.rs:9
Data Requirements (CSV Format)
The Gleaner expects a specific CSV format to successfully parse schemas. It uses a metadata-heavy header structure.
| Row Index | Content | Description |
|---|---|---|
| 1 | Header/Key | Must start with 'key' and contain field names or indices. |
| 2 | Description | Must start with '#' and contain field descriptions. |
| 3 | Types | Contains field types (e.g., str, int32, byte, bool). |
| 4+ | Data | Actual data rows begin from the fourth line. |
Sources: packages/noumenon-gleaner/src/main.rs:77-88
// Example CSV Format logic from main.rs
fn print_csv_format_guidance() {
eprintln!("key,0,1,2");
eprintln!("#,Name,Level,IsActive");
eprintln!("int32,str,byte,bool");
eprintln!("1,\"Test\",10,true");
}
Sources: packages/noumenon-gleaner/src/main.rs:79-83
Build and Deployment
The package uses a hybrid build process involving Cargo and Node.js scripts.
- Cargo Build: Compiles the Rust source code into a release executable.
- Postbuild Script: A Node.js script (
scripts/postbuild.js) handles platform-specific tasks:- Creates a
bin/directory. - Moves the executable from
target/release/tobin/. - Sets appropriate permissions (755) on Unix systems.
- Generates a
.cmdbatch file for Windows compatibility.
- Creates a
Sources: packages/noumenon-gleaner/package.json:11, packages/noumenon-gleaner/scripts/postbuild.js:12-42
Development Commands
| Command | Action |
|---|---|
npm run build | Runs cargo build --release followed by postbuild.js. |
npm run item | Runs the gleaner with a fixture (Item.csv) and outputs to output/schemas.ts. |
cargo test | Executes the Rust test suite. |
Sources: packages/noumenon-gleaner/package.json:10-14, AGENTS.md:55
Error Handling
The Gleaner provides detailed error messages and recovery suggestions. If a required file is missing, the tool invokes analyze_missing_files to suggest specific CSV filenames that need to be created based on the input file's references.
Sources: packages/noumenon-gleaner/src/main.rs:17-23, packages/noumenon-gleaner/src/main.rs:64-74
Summary
Noumenon Gleaner provides a robust mechanism for synchronizing data structures between raw CSV sources and TypeScript-based applications. By enforcing a strict header format in CSVs and automating interface generation, it reduces the risk of runtime errors in the Noumenon data search service.