Logging & Telemetry

Relevant source files

The following files were used as context for generating this wiki page:

• crates/logging/src/lib.rs
• crates/logging/src/config.rs
• crates/logging/src/macros.rs
• crates/logging/src/request_id.rs
• crates/error/src/middleware.rs
• IMPLEMENTATION_PLAN.md

The Logging & Telemetry system in Horizon provides a structured infrastructure for capturing system events, performance metrics, and audit trails. Built on the tracing ecosystem, it enables high-performance asynchronous logging with support for multiple output formats including JSON for production environments and human-readable formats for development.

The system is designed to provide end-to-end visibility across the distributed architecture, utilizing correlation IDs (Request IDs) to trace operations from the API server through to the database and external AI providers. This infrastructure is critical for the "Integrity" strategic goal of the project, ensuring complete audit trails and data quality assessment.

Sources: crates/logging/src/lib.rs:1-6, IMPLEMENTATION_PLAN.md:148-151, IMPLEMENTATION_PLAN.md:27-32

System Architecture

The logging architecture is centralized in the logging crate and integrated throughout the workspace. It utilizes a subscriber-layer pattern where different filters and formatters are composed based on the runtime configuration.

The diagram shows the flow from configuration to the final output targets via the tracing registry.
Sources: crates/logging/src/config.rs:60-72, crates/logging/src/lib.rs:24-30

Core Components

Component	Description
LoggingConfig	Handles environment-based configuration and building the tracing subscriber.
RequestId	A unique identifier (CUID2) used for log correlation across service boundaries.
tracing-subscriber	The underlying engine for managing log layers and event distribution.
tracing-appender	Manages non-blocking file output with hourly rolling policies.

Sources: crates/logging/src/config.rs:13-30, crates/logging/src/request_id.rs:1-10, crates/logging/Cargo.toml:17-20

Configuration and Initialization

The system supports three-tier configuration: programmatic defaults, environment variables, and configuration files. It specifically looks for RUST_LOG for levels and HORIZON_LOG_FORMAT for output styles.

Logging Configuration Options

Option	Type	Default	Description
level	String	"info"	Log level (trace, debug, info, warn, error).
format	String	"json"	Output format: json, pretty, or compact.
log_file	Option<String>	None	Optional path for rolling file logs.
include_timestamp	bool	true	Whether to include timestamps in log entries.
environment	String	"development"	The runtime environment (e.g., testing, production).

Sources: crates/logging/src/config.rs:13-30, crates/logging/src/config.rs:43-57

Initialization Process

The initialization involves building a subscriber registry that includes level filtering and formatting layers. In production environments, it defaults to JSON formatting with UTC RFC-3339 timestamps.

pub fn init(
    level: &str,
    format: &str,
    log_file: Option<&str>,
) -> Result<(), tracing::subscriber::SetGlobalDefaultError> {
    let config = LoggingConfig::from_env(level, format, log_file);
    let subscriber = config.build();
    tracing::subscriber::set_global_default(subscriber)?;
    info!(level = %level, format = %format, "Logging initialized");
    Ok(())
}

Sources: crates/logging/src/lib.rs:24-34, crates/logging/src/config.rs:75-78

Structured Logging Macros

Horizon provides a set of domain-specific macros to ensure consistent metadata across different system modules. These macros automatically inject context such as request IDs, durations, and target categories.

Request Correlation

Macros like info_with_request! and error_with_request! require a RequestId and use the % sigil for display-based formatting of the identifier.

The sequence shows how the RequestID correlates events across different targets.
Sources: crates/logging/src/macros.rs:11-40, crates/logging/src/macros.rs:55-68

Measurement and Scoping

The system includes utilities for measuring block execution time and creating scoped spans.

• measure_duration!: Wraps a block of code, calculates elapsed time using std::time::Instant, and logs the result at the DEBUG level.
• span_duration!: Creates a tracing::info_span that includes a start time for long-running asynchronous operations.

Sources: crates/logging/src/macros.rs:125-150, crates/logging/src/macros.rs:155-168

Correlation and Context

Correlation is achieved through the RequestId type, which encapsulates a CUID2 (Collision-resistant Unique Identifier). This ID is extracted from headers or generated at the start of a request flow.

RequestId Logic

The system attempts to extract existing IDs from headers via extract_from_headers and try_from_header. If no valid ID is present, it generates a new one. This ensures that a single operation can be tracked across the entire stack.

Middleware Integration

Telemetry is integrated into the Axum web framework via specialized middleware components found in the error crate.

• RequestLogger: A middleware that captures API request details while allowing exclusions for high-frequency endpoints like /health and /ready.
• ErrorHandler: Converts internal AppError types into structured ApiResponse objects, logging the error details based on the include_details flag. If a rate limit is hit, the telemetry captures the retry_after duration.

Sources: crates/error/src/middleware.rs:36-60, crates/error/src/middleware.rs:72-85

Conclusion

The Logging & Telemetry system provides the foundation for observability in Horizon. By combining structured JSON logging, correlation IDs, and domain-specific macros, it enables developers to perform precise debugging and system monitoring. The separation of concerns between the logging crate (infrastructure) and the error crate (middleware integration) ensures that telemetry is consistently captured across both the API server and the background workers.