API Backpressure & Load Shedding
Relevant source files
The following files were used as context for generating this wiki page:
API Backpressure and Load Shedding in Infrahub protect the system from saturation by managing request volume based on priority. The system implements a per-worker admission layer in the FastAPI backend that sheds load when the database or worker capacity is exceeded. This ensures that high-priority interactive traffic remains responsive even when background tasks or massive queries overload the server.
Sources: dev/knowledge/backend/api-backpressure.md, specs/archive/ifc-2886-priority-api-backpressure/plan.md:9-12
System Architecture
The admission layer consists of several components working together to evaluate and throttle incoming requests. It is registered as the outermost ASGI middleware to ensure load shedding occurs before any significant handler work begins.
Core Components
- AdmissionMiddleware: The entry point for all HTTP requests. It parses priority headers and communicates with the controller to decide if a request should proceed.
- AdmissionController: The central decision engine that wires together the slot pool and the CoDel controllers.
- PrioritySlotPool: A concurrency primitive that bounds the number of requests actively being processed. It maintains separate queues for different priority classes.
- CoDelController: An implementation of the Controlled Delay (CoDel) algorithm. It tracks "sojourn time"—the time a request spends waiting in the queue—and adaptively sheds traffic when delays exceed configured targets.
Sources: specs/archive/ifc-2886-priority-api-backpressure/plan.md:14-20, backend/infrahub/api/admission/controller.py:17-30
Request Admission Flow
The following diagram illustrates the lifecycle of a request through the admission layer:
The diagram shows how the middleware filters requests, parses priority, and uses the controller to evaluate backstop limits and CoDel shedding before allowing the application to process the request.
Sources: specs/archive/ifc-2886-priority-api-backpressure/plan.md:60-68, specs/archive/ifc-2886-priority-api-backpressure/tasks.md:65-71
Priority Management
Infrahub uses a cooperative priority model where clients declare their intent. Priority is determined by the X-Priority header.
Priority Classes
| Class | Description | Default Target Multiplier |
|---|---|---|
| High | Interactive UI traffic and critical operations. | 4.0x |
| Normal | Standard API requests and typical operations. | 1.0x |
| Low | Background tasks, bulk imports, and non-interactive scripts. | 1.0x |
- If the
X-Priorityheader is missing or invalid, the system defaults to Normal priority. - The admission gradient ensures that Low priority traffic is shed first, followed by Normal, while preserving High priority capacity.
Sources: specs/archive/ifc-2886-priority-api-backpressure/plan.md:9-12, specs/archive/ifc-2886-priority-api-backpressure/tasks.md:43-50, backend/infrahub/api/admission/controller.py:35-42
Concurrency and Capacity Derivation
The system derives its capacity from the environment rather than using arbitrary magic numbers. The max_concurrency for the slot pool is typically derived from the Neo4j connection pool size.
- Derivation Formula:
max(1, int(pool_size * factor)) - Signal Binding: Sojourn time only increases when in-flight requests reach the
max_concurrencylimit, signaling that the database connection pool is saturated.
Sources: specs/archive/ifc-2886-priority-api-backpressure/tasks.md:60-62, specs/archive/ifc-2886-priority-api-backpressure/plan.md:73-77
Configuration Options
Administrators can tune the backpressure system using environment variables prefixed with INFRAHUB_API_.
| Setting | Type | Default | Description |
|---|---|---|---|
backpressure_enabled | Boolean | True | Master kill-switch for the admission layer. |
backpressure_codel_target_seconds | Float | 0.005 | The acceptable queuing delay (sojourn time). |
backpressure_codel_interval_seconds | Float | 0.1 | The window over which sojourn time is averaged for CoDel. |
backpressure_high_target_multiplier | Float | 4.0 | Multiplier applied to the target for High priority requests. |
backpressure_retry_after_seconds | Integer | 1 | Value for the Retry-After header sent with 429 responses. |
backpressure_max_concurrency_factor | Float | 1.0 | Multiplier for the derived DB connection pool size. |
Sources: specs/archive/ifc-2886-priority-api-backpressure/tasks.md:43-50
Observability and Metrics
The admission layer exports Prometheus metrics via the /metrics endpoint to provide visibility into load shedding performance.
| Metric Name | Type | Description |
|---|---|---|
infrahub_admission_offered_total | Counter | Total requests offered to the admission layer by priority. |
infrahub_admission_admitted_total | Counter | Requests successfully admitted for processing. |
infrahub_admission_rejected_total | Counter | Requests shed, labeled by reason (codel or backstop). |
infrahub_admission_in_flight | Gauge | Number of requests currently executing in the worker. |
infrahub_admission_waiters | Gauge | Number of requests waiting in the priority queues. |
infrahub_admission_sojourn_seconds | Histogram | Distribution of time spent waiting for a slot. |
infrahub_admission_max_concurrency | Gauge | The calculated maximum concurrency limit for the worker. |
Sources: specs/archive/ifc-2886-priority-api-backpressure/tasks.md:65-71, specs/archive/ifc-2886-priority-api-backpressure/plan.md:12-13
Load Shedding Logic (CoDel)
The AdmissionController uses the CoDel algorithm to manage shedding. Unlike simple rate limiting, CoDel focuses on delay. If sojourn time remains above the target for a full interval, the controller enters a "dropping" state. It exits this state immediately when a single sample falls below the target.
# backend/infrahub/api/admission/controller.py:35-48
def admit(self, priority: Priority) -> AdmissionDecision:
self.metrics.offered_total.labels(priority=priority.name.lower()).inc()
# 1. Backstop check
if self.slot_pool.waiter_count(priority) >= self.settings.backpressure_backstop_max_waiters:
return Rejected(reason="backstop", retry_after=self.settings.backpressure_retry_after_seconds)
# 2. Acquire slot and measure sojourn
with self.slot_pool.acquire(priority) as sojourn:
# 3. CoDel evaluation
if self.codel_controllers[priority].should_drop(sojourn):
return Rejected(reason="codel", retry_after=self.settings.backpressure_retry_after_seconds)
return Admitted(slot=sojourn.slot)
Sources: backend/infrahub/api/admission/controller.py:35-48, specs/archive/ifc-2886-priority-api-backpressure/tasks.md:73-77
Load shedding ensures system stability under stress. By shedding requests at the edge with a 429 Too Many Requests status, Infrahub prevents resource exhaustion in the core application logic and the Neo4j database. The priority-aware gradient guarantees that background operations like sync_with_git or large exports do not disrupt the user experience in the web UI.