Telemetry & Observability

Relevant source files

The following files were used as context for generating this wiki page:

• src/telemetry.rs
• src/metrics.rs
• src/stream_metrics.rs
• src/stream_metrics/tests.rs
• ARCHITECTURE.md
• src/main.rs
• src/proxy.rs

The Telemetry and Observability system in shunt provides structured logging, distributed tracing, and real-time metrics collection to ensure operational visibility into the LLM gateway. It is designed to be opt-in and flexible, supporting local console logging as well as remote export via OpenTelemetry (OTel) and Sentry.

This system monitors the entire request lifecycle—from initial buffering and routing to upstream dispatch and streaming response delivery—allowing operators to track latency, request counts, and error rates across different providers and models.

Sources: ARCHITECTURE.md:104-108, src/telemetry.rs

Architecture and Data Flow

The observability stack is initialized at process startup within main.rs and persists throughout the application lifecycle. It utilizes tracing as the primary instrumentation facade, with multiple subscribers (layers) for different export targets.

The diagram shows the initialization of telemetry from the main entry point and how request-level spans from the proxy flow into the unified tracing subscriber.

Sources: src/main.rs, src/telemetry.rs, ARCHITECTURE.md:65-71

Instrumentation Layers

The telemetry system organizes observability into three distinct pillars:

1. Tracing: Captures the flow of execution. Each request is wrapped in a proxy_request span containing metadata such as the HTTP method, path, and an optional session_id.
2. Metrics: Aggregates numerical data such as request counters and latency histograms.
3. Error Tracking: Integrates with Sentry for automated exception and trace capture.

Sources: src/telemetry.rs, ARCHITECTURE.md:104-108, src/proxy.rs:43-50

Metrics Collection

Metrics are handled primarily through the metrics.rs module and specialized stream monitors. These metrics track both gateway-level performance and provider-specific behavior.

Request and Stream Metrics

shunt distinguishes between standard request metrics and high-granularity stream metrics. Stream metrics are particularly important for LLM workloads where "Time to First Token" (TTFT) and throughput are critical.

Component	Description	Key Metrics
Request Metrics	Global counters for the proxy.	shunt_requests_total, shunt_request_duration_seconds
Stream Metrics	Per-chunk and per-token analytics.	stream_chunks_total, stream_latency_ms, tokens_per_second

Sources: src/metrics.rs, src/stream_metrics.rs

Stream Metrics Logic

The StreamMetrics struct tracks the timing of Server-Sent Events (SSE) from upstream providers. It records the interval between the request start and the first received chunk, as well as subsequent inter-chunk intervals.

This diagram illustrates how metrics are captured during a streaming LLM response.

Sources: src/stream_metrics.rs, src/proxy.rs

Configuration and Initialization

Telemetry is configured via the shunt.toml or environment variables. The initialization logic in src/telemetry.rs checks for the presence of OTel or Sentry configuration before spawning background export threads.

Key Telemetry Functions

Function	File	Description
init_telemetry()	src/telemetry.rs	Initializes tracing, OTel exporters, and Sentry.
proxy_request span	src/proxy.rs	Defines the instrumentation context for every inbound request.
record_stream_metrics()	src/stream_metrics.rs	Updates internal registries with data from a completed stream.

Sources: src/telemetry.rs, src/proxy.rs, src/stream_metrics.rs

Implementation Detail: The Tracing Span

The gateway wraps every request in a span to ensure all logs emitted during that request are correlated.

// src/proxy.rs:43-50
let span = tracing::info_span!(
    "proxy_request",
    method = %req.method(),
    path = %req.uri().path(),
    session_id = tracing::field::Empty,
);

Sources: src/proxy.rs:43-50

Observability Invariants

• Non-Blocking: Telemetry export (OTLP/Sentry) occurs on dedicated SDK threads to prevent observability overhead from impacting request latency.
• Opt-in Export: While basic logging is standard, OTel and Sentry exports are only active if the operator explicitly configures them.
• Static Initialization: Telemetry exporters are built once at startup. Unlike the routing configuration, telemetry settings are not reloaded during hot-reloads to maintain stable connection pools to observability backends.

Sources: ARCHITECTURE.md:104-108, src/telemetry.rs

Summary

The Telemetry & Observability system in shunt provides a robust foundation for monitoring a high-performance LLM gateway. By combining structured tracing with specialized streaming metrics, it enables developers to diagnose issues ranging from simple routing errors to complex upstream latency regressions.

Sources: ARCHITECTURE.md:104-108, src/stream_metrics.rs