Metrics & Telemetry

Relevant source files

The following files were used as context for generating this wiki page:

• internal/metrics/metrics.go
• internal/server/logging_middleware.go
• internal/server/service.go
• internal/server/dynamic_redirects.go
• internal/cmd/cache_stats.go
• ROADMAP.md

The kamal-proxy project implements a robust observability stack designed to provide real-time insights into traffic patterns, cache performance, certificate management, and system health. The primary purpose of this system is to enable operators to monitor proxy performance, debug traffic issues, and maintain high availability through data-driven decisions.

The telemetry architecture is composed of three main pillars: structured logging via the slog library, high-resolution metrics exported through Prometheus, and specialized CLI tools for inspecting internal state such as cache statistics. This data is scoped at both the global proxy level and the individual service level, allowing for granular analysis of specific application workloads.

Metrics Architecture

The metrics system is built around a central tracker interface, which defines the events that can be recorded across the application. The default implementation uses the Prometheus client library to export gauges, counters, and histograms.

Core Request Metrics

Request telemetry captures the volume, latency, and concurrency of traffic flowing through the proxy.

• HTTP Requests Total: A counter labeled by service, HTTP method, and status code.
• HTTP Duration Seconds: A histogram recording request latency, using standard Prometheus buckets.
• In-flight Requests: A gauge reflecting the current number of active requests being processed per service.

Sources: internal/metrics/metrics.go:81-118, internal/server/logging_middleware.go:70-108

Performance & Specialized Metrics

The proxy tracks specialized events related to its advanced features:

Metric Category	Source	Description
Response Cache	cache_events_total	Outcomes including hit, miss, stale, and coalesced.
Certificate Stats	certificate_expiry_timestamp	Unix timestamp of managed certificate expiration.
Dynamic Redirects	dynamic_redirect_map_size	Size of host-scoped redirect maps (hosts/rules).
Denials	denials_total	Requests refused by IP allow/deny lists or User-Agent rules.

Sources: internal/metrics/metrics.go:120-179, ROADMAP.md:18-20

Structured Logging

The proxy utilizes structured logging to provide detailed context for every request. This is implemented via a specialized LoggingMiddleware that wraps the HTTP handler chain.

Log Context Attributes

Every log entry for a request includes a rich set of attributes:

• Standard Identifiers: request_id, service, target, method, path.
• Network Details: client_addr, remote_addr (resolving X-Forwarded-For), scheme, proto.
• Payload Info: req_content_length, resp_content_length, status.
• Tracing Context: trace_id, span_id, and trace_flags are included if W3C traceparent headers are present.

Sources: internal/server/logging_middleware.go:76-102, ROADMAP.md:92-96

Telemetry Configuration

Operators can control the behavior of metrics and logs through service-level and global options.

Option	Type	Description
--exclude-metrics-path	[]string	Specific paths (like /up) to exclude from metrics tracking.
--log-format	string	Toggles between json and text output.
--trace-context	string	Controls OTel traceparent propagation (off, propagate, generate).

Sources: internal/server/service.go:121, internal/server/service.go:175, ROADMAP.md:92-95

Cache Telemetry & Statistics

In addition to Prometheus metrics, the proxy provides a dedicated RPC-based mechanism for inspecting the response cache state through the kamal-proxy cache stats command.

Cache Stats Report

The CacheStats data structure provides a comprehensive view of the cache store:

• Store Type: Identifies if the store is in-process LRU or a shared Redis.
• Entry Sizing: Total entries and total bytes stored.
• Eviction Health: Tracks "fresh" vs "stale" evictions. High fresh evictions indicate the cache budget (--cache-memory-size) is too small.
• Oversized Refusals: Count of responses that exceeded the --cache-max-body limit.

Sources: internal/cmd/cache_stats.go:50-78, internal/metrics/metrics.go:149-153

Summary

The Metrics & Telemetry system in kamal-proxy is designed for high-performance observability. By combining the tracker interface for Prometheus metrics with a comprehensive LoggingMiddleware, the proxy ensures that every request, cache event, and certificate renewal is visible to operators. The inclusion of trace context propagation and configurable metric exclusions allows the proxy to integrate seamlessly into modern distributed tracing and monitoring environments.