Resilience Policies & Deadlines
Relevant source files
The following files were used as context for generating this wiki page:
The Resilience and Deadlines system in Forze provides a robust framework for fault tolerance, production hardening, and time-bound operation execution. It allows developers to compose structured resilience strategies—such as retries, circuit breakers, and bulkheads—while enforcing strict invocation time budgets to ensure system stability and responsiveness under load.
This system is integrated deeply into the Forze execution engine, allowing for both declarative application of policies via operation registries and imperative execution around specific calls. By combining these policies with dead-line propagation and graceful shutdown mechanisms, Forze facilitates the deployment of reliable "fleet" posture applications.
Resilience Policies
Resilience in Forze is managed through ResiliencePolicy objects, which consist of an ordered stack of strategies. These strategies process requests from outer to inner layers, typically following a sequence such as: Rate Limit → Bulkhead → Circuit Breaker → Retry → Per-attempt Timeout.
Policy Composition & Built-ins
Forze ships with pre-configured policies for common scenarios:
occ: Specifically handles retries for concurrency-related failures.transient: Designed for infrastructure-level failures, featuring a 30-second per-attempt timeout and retries.
Sources: skills/forze-resilience-deadlines/SKILL.md:14-18
Resilience Strategies
Individual strategies provide specific fault-tolerance mechanisms.
| Strategy | Type | Description |
|---|---|---|
| Bulkhead | Fixed/Adaptive | Limits concurrent executions to prevent resource exhaustion. Adaptive versions use AIMD based on observed latency. |
| Circuit Breaker | State-based | Prevents calls to a failing downstream by "opening" after a failure threshold is reached. |
| Adaptive Throttle | Probabilistic | Sibling to the breaker for degraded downstreams; sheds traffic locally based on downstream absorption rates. |
| Retry | Iterative | Re-runs operations on "retryable" error kinds: concurrency, infrastructure, and throttled. |
| Hedge | Concurrent | Races a secondary "hedged" call against a slow primary for idempotent reads. |
Sources: skills/forze-resilience-deadlines/SKILL.md:29-57
Execution Logic
The ResilienceExecutor is responsible for running operations under a specified policy. It resolves the policy by name and applies it to the provided callable.
The diagram above shows the typical logical flow through a composed resilience policy stack.
Sources: src/forze/application/execution/resilience/executor.py:12-45, skills/forze-resilience-deadlines/SKILL.md:14-22
Invocation Deadlines
Deadlines represent the total time budget allocated to an operation. Unlike per-attempt timeouts, a deadline is non-retryable; once the budget is spent, the operation raises a terminal timeout error.
Deadline Propagation
Forze supports cross-service deadline propagation. The outbound HTTP adapter forwards the remaining budget via the X-Forze-Deadline-Budget header. On the receiving end, the InvocationMetadataMiddleware in FastAPI can be configured to honor this header, ensuring the budget is tightened across service boundaries.
Sources: skills/forze-resilience-deadlines/SKILL.md:67-73, skills/forze-fastapi-interface/SKILL.md:65-68
Implementation Detail: remaining_time()
The execution context provides a remaining_time() function. Ports and handlers use this to determine the current available budget and derive per-call timeouts for underlying drivers.
# Declarative deadline on a registry
registry.bind("orders.create").with_deadline(timedelta(seconds=5)).finish().freeze()
# Imperative budget check
budget = ctx.inv_ctx.remaining_time()
Sources: skills/forze-resilience-deadlines/SKILL.md:61-65, src/forze/application/execution/context/deadline.py:8-15
Graceful Shutdown & Fleet Posture
The resilience system extends to the lifecycle of the application, particularly regarding how replicas (the "fleet") behave during deployment and shutdown.
Draining and Readiness
When a runtime starts its shutdown sequence:
- Draining: New top-level invocations are rejected with a
throttlederror (code:"draining"). - Graceful Wait: In-flight operations are given a
drain_timeout(default 10s) to complete. - Readiness: The
/readyzroute switches from200 OKto503 Service Unavailable, signaling load balancers to stop routing traffic to the instance.
Sources: skills/forze-resilience-deadlines/SKILL.md:77-84, skills/forze-fastapi-interface/SKILL.md:61-63
Fleet Validation
Setting DeploymentProfile.FLEET enables strict validation at assembly time. It ensures that any lifecycle step that mutates_shared_state is properly guarded (e.g., via a distributed lock) to prevent race conditions when multiple replicas start simultaneously.
Sources: skills/forze-resilience-deadlines/SKILL.md:88-95
Summary
Resilience in Forze is a multi-layered approach that combines declarative policy stacks (Bulkheads, Breakers, Retries) with strict, transmissible invocation deadlines. By leveraging the ResilienceExecutor and standardizing on CoreException error kinds, the framework ensures that backend services can gracefully handle transient failures and sustained congestion while providing clear signals to infrastructure for load balancing and scaling.
Sources: src/forze/application/execution/resilience/executor.py, skills/forze-resilience-deadlines/SKILL.md