Data Management & Flow

Relevant source files

The following files were used as context for generating this wiki page:

• crates/cli/src/main.rs
• crates/error/src/response.rs
• crates/error/src/codes.rs
• docker-compose.yml
• IMPLEMENTATION_PLAN.md
• README.md
• AGENTS.md

Horizon utilizes a multi-tiered data management strategy to provide comprehensive visibility into IT infrastructure. The system manages data through automated collection via lightweight agents, structured persistence in a relational database, and high-speed caching for session and performance optimization. The data flow architecture ensures integrity through audit trails, encryption at rest, and a standardized API response format for all consumers.

Sources: README.md:16-25, IMPLEMENTATION_PLAN.md:25-30

Persistence Layer Architecture

Horizon's data persistence is built on a primary relational database and a secondary caching layer. This infrastructure is orchestrated using Docker for development and production environments, ensuring consistent data environments across different stages of the lifecycle.

Primary Data Store: PostgreSQL

PostgreSQL serves as the primary Configuration Management Database (CMDB), storing core entities such as assets, user accounts, and security configurations. The system uses Sea-ORM for entity management and migration tracking.

Caching and Session Management: Redis

Redis is utilized for high-performance data operations, specifically handling session management, token blacklisting, and rate-limiting middleware.

Data Models and Schema Design

The system organizes data into several core modules, each with dedicated schema definitions. This modular approach allows for granular control over different IT asset management functions.

Core Data Entities

Module	Key Entities	Purpose
Identity	Users, Teams, Roles	RBAC and authentication management.
Assets	Asset Types, Assets, Relationships	Inventory of hardware and dependency mapping.
Software	Products, Versions, Installations	Tracking software lifecycle and licenses.
Security	Vulnerabilities, CVEs, Baselines	Risk assessment and compliance tracking.
Network	Subnets, Interfaces, Flows	Topology mapping and traffic analysis.

Sources: IMPLEMENTATION_PLAN.md:160-250, IMPLEMENTATION_PLAN.md:310-400

Database Configuration

The CLI and Server use environment variables to construct the DATABASE_URL required for connecting to the persistence layer.

pub fn build_database_url(config: &DatabaseConfig) -> String {
    format!(
        "postgres://{}:{}@{}:{}/{}?sslmode={}",
        config.username, config.password, config.host, config.port, config.database,
        config.ssl_mode
    )
}

Sources: crates/cli/src/main.rs:56-62

API Data Flow and Standardization

Data exiting the system via the REST API follows a strict, standardized format to ensure consistency across web, desktop, and CLI consumers. This is handled by the ApiResponse type and its builder.

Success and Error Structures

All API responses are wrapped in a generic container that includes a status tag (success or error), the payload data, and metadata.

Response Metadata

Metadata provides contextual information about the data transaction, including pagination for collection resources and request correlation IDs.

Field	Type	Description
request_id	String	Unique identifier for tracing the request flow.
page	u64	Current page number (1-indexed).
total_items	u64	Total count of items available in the dataset.
response_time_ms	u64	Time taken to process the request.

Sources: crates/error/src/response.rs:25-50, crates/error/src/response.rs:95-105

Error Handling and Classification

Horizon implements a structured error handling system where every error is classified by category and assigned a specific code. This facilitates better data integrity tracking and client-side error resolution.

Error Categories

Errors are categorized to help the system decide whether to log the event or expose details to the end-user.

Category	Status Code	Logging	Details Exposure
Validation	422	No	Yes
Authentication	401	No	No
Database	500	Yes	No
Client	400/404/409	No	Yes

Sources: crates/error/src/codes.rs:15-150

Data Collection Flow

The primary data entry point for automated inventory is the Horizon Agent. It follows a structured path from local system discovery to server-side ingestion.

1. Discovery: Agent collects hardware (CPU, Memory, Storage) and software (installed packages, services) info.
2. Normalization: Data is structured into a platform-agnostic SystemInfo struct.
3. Transmission: Agent sends Protobuf-encoded messages via mTLS to the API server.
4. Ingestion: Server validates signatures, decrypts the payload, and updates the asset history and current status in PostgreSQL.

Sources: IMPLEMENTATION_PLAN.md:280-305, AGENTS.md:50-80, IMPLEMENTATION_PLAN.md:550-570

Conclusion

Data Management & Flow in Horizon is designed for high reliability and security. By combining Sea-ORM for structured relational data, Redis for transient state management, and a standardized API response framework, the system ensures that IT infrastructure data is accurately captured, securely stored, and consistently presented to all interfaces.