Graph Rebuild & Recovery

Graph Rebuild & Recovery refers to the system's ability to reconstruct the asset relationship network and recover from failures or state loss. This module ensures that the financial asset graph remains consistent and durable across system restarts or process terminations. The architecture prioritizes idempotency and deterministic behavior during rebuild operations to prevent state corruption.

The system uses specialized executors to decouple the rebuild logic from other internal engines, such as reconciliation. It relies on durable persistence and distributed hosting protocols to manage multi-instance behavior and prevent stale owners from mutating state after a recovery event.

Sources: CHANGELOG.md, GEMINI.md:Section 5

Rebuild Architecture

The rebuild system is organized around the RebuildExecutor, which handles the execution logic for graph reconstruction. This component is decoupled from the ReconciliationEngine to ensure modularity and specialized error handling. The architecture enforces resource safety through context managers and absolute disposal pathways.

Rebuild Workflow

The rebuild process follows a strict execution path that includes failure detection and recovery gating. If the system detects a partial failure or a process termination, the recovery gate prevents invalid state transitions.

The diagram shows the sequential flow from initiation to successful state persistence, highlighting the role of the Recovery Gate in halting unsafe operations.
Sources: src/logic/rebuild_executor.py, CHANGELOG.md, GEMINI.md:Section 3

Failure Detection and Recovery Gating

The RebuildFailureDetection and RecoveryGate modules provide the logic for identifying system inconsistencies. The system must remain safe against partial failures, such as network drops or process terminations.

Operational Constraints

Recovery operations must adhere to specific architectural invariants:

• Idempotency: All rebuild operations must produce the same result regardless of how many times they run.
• Concurrency Control: Lock context managers use a non-configurable Maximum Bound (TTL) of 300 seconds.
• Deterministic Back-off: Acquisition attempts use a 30-second wait ceiling before throwing a LockAcquisitionTimeout.

Sources: src/logic/rebuild_failure_detection.py, src/logic/recovery_gate.py, GEMINI.md:Section 3

Persistence and Disaster Recovery

Durable graph persistence is a gating dependency for system promotion and disaster recovery. The system requires durable persistence for restarts, promotions, and recovery from multi-instance conflicts.

Distributed Hosting Rules

In distributed environments, the recovery system prevents "split-brain" scenarios where multiple instances attempt to rebuild the graph simultaneously:

1. Stale owners must never mutate state after a restart.
2. Rebuild cancellation and lock-loss behavior must be deterministic.
3. SQLite compatibility is preserved for local development while supporting enterprise persistence.

The sequence diagram illustrates the locking mechanism that ensures only one instance performs a rebuild at a time, protecting the durability of the storage.
Sources: GEMINI.md:Section 5, src/logic/rebuild_recovery.py

Recovery Implementation Details

The implementation of RebuildExecutor ensures that all file handles and sessions are managed via context managers. This prevents resource leaks during the heavy I/O operations required to rebuild large asset relationship datasets.

Failure Modes

When a change fails, the system follows specific failure modes to ensure a clean state:

• File Stream Interruptions: The system closes handles immediately and reverts to the last known durable checkpoint.
• Database Connection Drops: The RecoveryGate halts active rebuilds until connectivity is restored and the lock is re-verified.

Sources: src/logic/rebuild_executor.py, GEMINI.md:Section 1, GEMINI.md:Section 3

Graph Rebuild & Recovery provides the necessary safeguards for the Financial Asset Relationship Database to handle complex failures. By enforcing idempotency, deterministic locking, and durable persistence, the system ensures that the network visualization and metrics remain accurate even after unexpected terminations.