Distributed Tracing Sampling
Relevant source files
The following files were used as context for generating this wiki page:
Distributed Tracing Sampling in the FLEXT ecosystem is a production performance optimization strategy that utilizes head-based sampling to reduce observability overhead. By making sampling decisions at the request entry point, the system maintains visibility into critical operations while minimizing the volume of telemetry data generated in high-traffic environments.
Sources: src/flext_observability/services/sampling.py:1-12, src/flext_observability/README.md:59-60
The sampling mechanism is integrated with the broader flext-observability suite, coordinating with FlextTracingService and context propagation utilities. It allows for deterministic sampling decisions based on correlation IDs and supports fine-grained overrides for specific services and operations to ensure high-value traces are always captured.
Sources: src/flext_observability/services/sampling.py:237-245, examples/README.md:158-185
Sampling Architecture
The sampling logic is encapsulated within the FlextObservabilitySampling class, which manages a singleton Sampler instance. This architecture ensures a consistent sampling state across the application lifecycle.
Component Interaction
The following diagram illustrates the relationship between the Sampler, the Observability Context, and the decision-making process.
The sampler retrieves the correlation_id from the global context to ensure that if a request is sampled, all related spans within that request's lifecycle are consistently treated.
Sources: src/flext_observability/services/sampling.py:237-245, tests/integration/test_phase_11_integration.py:27-33
Sampling Logic and Hierarchy
The Sampler evaluates sampling decisions using a specific hierarchy of rates. This allows developers to set broad defaults while providing precision for critical areas.
Rate Resolution Priority
- Operation Override: Highest priority; specific to a request type (e.g.,
POST /api/users). - Service Override: Medium priority; specific to a service (e.g.,
user-service). - Environment Default: Lowest priority; determined by the current environment setting (Dev, Staging, Production).
Sources: src/flext_observability/services/sampling.py:228-236
Deterministic Decision Process
To avoid inconsistent tracing (where some spans of a request are recorded but others are not), the system uses the correlation_id to generate a hash. If the context is unavailable, it falls back to a random decision.
Configuration and Environment Defaults
The sampler provides pre-configured rates based on common environment needs, which can be updated at runtime via the update_environment method.
Environment Default Rates
| Environment | Default Sampling Rate | Description |
|---|---|---|
development | 1.0 (100%) | Full visibility for debugging. |
staging | 0.5 (50%) | Balance between visibility and performance. |
production | 0.1 (10%) | Typical production rate to minimize overhead. |
Sources: src/flext_observability/services/sampling.py:73-77, src/flext_observability/services/sampling.py:165-181
Configuration Methods
| Method | Parameter | Description |
|---|---|---|
update_default_rate | rate: float | Overrides the environment-based default rate. |
update_operation_rate | operation: str, rate: float | Sets a specific rate for a named operation. |
update_service_rate | service: str, rate: float | Sets a specific rate for an entire service. |
update_environment | environment: str | Sets environment and resets default rate. |
Sources: src/flext_observability/services/sampling.py:108-212
Implementation Example
The following snippet demonstrates how to initialize the sampler and configure overrides during application startup, as seen in integration testing patterns.
from flext_observability import FlextObservabilitySampling
# 1. Access the singleton instance
sampler = FlextObservabilitySampling.active_sampler()
# 2. Configure for Production
sampler.update_environment("production")
# 3. Ensure a critical billing endpoint is always sampled
sampler.update_operation_rate("POST /api/billing/charge", 1.0)
# 4. Make a decision
if sampler.should_sample(operation="POST /api/billing/charge"):
# Proceed with trace creation
pass
Sources: src/flext_observability/services/sampling.py:42-60, tests/integration/test_phase_11_integration.py:29-32
Integration with Distributed Tracing
Sampling decisions are typically used before invoking tracing services. In a distributed workflow, the parent_trace_id is propagated alongside the sampling decision to maintain trace continuity.
Trace Correlation Flow
Distributed Tracing Sampling ensures that the FLEXT ecosystem remains performant by selectively capturing telemetry. By providing deterministic head-based sampling and environment-aware defaults, it allows developers to maintain deep visibility into production systems without the cost of 100% trace retention.
Sources: src/flext_observability/services/sampling.py:1-20, tests/README.md:144-150