Alternative Backends & OTLP Routing
Relevant source files
The following files were used as context for generating this wiki page:
Pydantic Logfire is built as an opinionated wrapper around OpenTelemetry (OTel), providing the flexibility to route telemetry data—including traces, metrics, and logs—to backends other than the primary Logfire platform. This is achieved through support for the OpenTelemetry Line Protocol (OTLP), allowing users to leverage existing infrastructure like the OTLP Collector or alternative OTel-compatible observability providers.
The system supports multiple simultaneous export pipelines, enabling "multi-routing" where data can be sent to both Logfire and an alternative backend (or multiple Logfire projects) during migrations or for redundant storage. This routing is controlled via SDK configuration and environment variables that interface with the underlying OpenTelemetry SDK.
Core Architecture and Routing Logic
Logfire manages routing by initializing specialized SpanProcessor, MetricReader, and LogRecordProcessor instances based on the provided configuration. When send_to_logfire is enabled, the SDK automatically configures OTLP exporters pointing to Logfire's regional endpoints. Users can extend this by providing additional_span_processors or by configuring standard OpenTelemetry environment variables.
The Export Pipeline Flow
The following diagram illustrates how telemetry signals are captured and routed through the Logfire SDK to various potential destinations.
Configuration of Alternative Backends
Routing to alternative backends can be configured through two primary methods: programmatic SDK calls and standard OpenTelemetry environment variables.
Environment Variable Routing
Logfire respects standard OpenTelemetry environment variables to route data to generic OTLP endpoints. If OTEL_EXPORTER_OTLP_ENDPOINT or signal-specific endpoints are detected, Logfire initializes additional exporters.
| Environment Variable | Description | Signal |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | The generic base URL for an OTLP-compatible backend. | All |
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT | Specific endpoint for trace data. | Traces |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | Specific endpoint for metric data. | Metrics |
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT | Specific endpoint for log data. | Logs |
OTEL_TRACES_EXPORTER | Set to 'otlp' to enable the OTLP span exporter. | Traces |
Sources: logfire/_internal/config.py:785-813, tests/test_configure.py:844-884
Programmatic Multi-Project Routing
The SDK supports routing data to multiple Logfire projects or alternative backends simultaneously by passing a list of tokens.
logfire.configure(
token=['token_project_a', 'token_project_b'],
additional_span_processors=[
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://my-collector:4318/v1/traces"))
]
)
Sources: logfire/_internal/config.py:382-390, tests/test_configure.py:1123-1135
OTLP Exporter Implementation Details
Logfire uses customized OTLP exporters to ensure reliability and performance when communicating with backends. These are primarily located in logfire._internal.exporters.otlp.py.
Specialized Exporter Components
- BodySizeCheckingOTLPSpanExporter: A wrapper that ensures payloads do not exceed backend limits before transmission.
- RetryFewerSpansSpanExporter: Logic that attempts to retry failed exports by reducing the batch size, which is useful when dealing with restrictive backend rate limits or payload size constraints.
- QuietExporters: Wrappers like
QuietSpanExporterandQuietLogExporterthat suppress noisy export errors to prevent logs from being flooded when a backend is unreachable.
Regional Routing via Tokens
Logfire automatically routes data to the correct geographic region based on the prefix of the provided write token. If a token is not provided but send_to_logfire is True, the SDK attempts to initialize a project and infer the region.
| Region Code | Base URL |
|---|---|
us | https://logfire-api.pydantic.dev |
eu | https://logfire-api-eu.pydantic.dev |
stagingus | https://logfire-us.pydantic.info |
stagingeu | https://logfire-eu.pydantic.info |
The get_base_url_from_token function parses the token using the PYDANTIC_LOGFIRE_TOKEN_PATTERN to determine these endpoints.
Sources: logfire/_internal/config.py:1015-1033, logfire/_internal/config_params.py:126-140
CLI Tooling for Routing
The Logfire CLI provides utilities to verify instrumentation and routing configuration without modifying code. The logfire run command can automatically detect and instrument installed packages, routing their telemetry to the configured backend.
logfire run --base-url <URL>: Overrides the default backend routing to point to a self-hosted instance or OTLP collector.logfire run --region <REGION>: Specifically routes data to a supported Logfire region (e.g.,usoreu).
Sources: logfire/_internal/cli/init.py:293-298, logfire/_internal/cli/run.py:73-86
Logfire's routing system ensures that while the SDK is optimized for the Logfire platform, it remains a "first-class citizen" in the OpenTelemetry ecosystem, allowing seamless integration with any OTLP-compatible backend or collector.