Noumenon Librarian
Relevant source files
The following files were used as context for generating this wiki page:
Noumenon Librarian is the data pipeline orchestrator for the Project Noumenon ecosystem. Its primary role is to manage and coordinate the flow of data required for the project's search services, specifically focusing on data extraction and schema management. It serves as the connective tissue between raw data sources and the structured data used by the Noumenon Frontend.
Sources: packages/noumenon-librarian/README.md, apps/noumenon-front/README.en.md
System Architecture
Noumenon Librarian functions as an orchestrator within a monorepo architecture. It leverages the noumenon-gleaner package to perform low-level data extraction tasks and utilizes the Effect ecosystem for robust functional programming patterns.
Core Dependencies
| Dependency | Purpose |
|---|---|
noumenon-gleaner | Internal workspace tool used for extracting schemas and data from CSV files. |
Effect (@effect/*) | Provides the functional foundation for concurrency, SQL integration, and platform-specific operations. |
Bun | The primary runtime and package manager used for execution and script management. |
Sources: packages/noumenon-librarian/package.json:11-18, CLAUDE.md:12-16
Data Pipeline Workflow
The Librarian orchestrates a multi-step process to transform raw gaming data into a structured format suitable for the search service. The following diagram illustrates the high-level data flow between the Librarian and its components:
The Librarian is responsible for loading raw data, executing the extraction tool, and ensuring the resulting schemas are correctly distributed within the project structure.
Sources: packages/noumenon-librarian/README.md:5-9, packages/noumenon-gleaner/src/main.rs:14-31
Integration with Noumenon Gleaner
The Librarian relies heavily on noumenon-gleaner, a Rust-based CLI tool. While the Librarian handles orchestration, the Gleaner performs the actual parsing of CSV files to generate TypeScript interfaces.
Gleaner Execution Logic
The Gleaner tool follows a specific sequence when invoked by the Librarian:
- Configuration Parsing: It accepts input and output paths via CLI arguments.
- Schema Building: It analyzes the CSV structure (specifically the first three rows containing keys, descriptions, and types).
- TypeScript Generation: It converts the analyzed schema into TypeScript interfaces saved to the specified output path.
Operational Commands
The Librarian provides standard scripts for preparation and execution.
| Command | Action |
|---|---|
bun run prepare | Executes scripts/prepare.ts to set up the environment. |
bun run start | Executes main.ts to begin the orchestration pipeline. |
Sources: packages/noumenon-librarian/package.json:6-9
Project Noumenon Context
Within the broader scope of the repository, Noumenon Librarian is one of several specialized packages working toward the goal of creating a Final Fantasy XIV data search service.
- noumenon-front: The user-facing search service (Astro/React).
- noumenon-gleaner: The Rust-based data extractor.
- noumenon-librarian: The pipeline orchestrator.
Sources: README.md:16-25, apps/noumenon-front/README.en.md:3-5
Conclusion
Noumenon Librarian serves as the central automation hub for Project Noumenon's data needs. By orchestrating the Rust-based Gleaner tool and managing the lifecycle of data schemas, it ensures that the frontend search service has access to type-safe and accurate data structures derived directly from game resources.
Sources: packages/noumenon-librarian/README.md, packages/noumenon-gleaner/README.md