Disaster Recovery
Relevant source files
The following files were used as context for generating this wiki page:
Disaster Recovery (DR) for the X3 ATOMIC STAR project encompasses a suite of battle-tested procedures, runbooks, and verification exercises designed to recover from catastrophic failures. The scope of DR includes handling chain halts, consensus failures, state corruption, and bridge exploits, ensuring the network can maintain or restore asset safety and service continuity under extreme pressure. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:1-12, crates/x3-relayer/PHASE_13F_MASTER_INDEX.md:10-25
The system relies on a structured Incident Response framework, classifying events by severity (S0 to S3) and defining specific roles such as the Incident Commander and Technical Lead to manage the lifecycle of a crisis. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:21-50
Incident Classification and Severity
Incidents are classified based on their impact on network stability and asset safety. Severity S0 represents catastrophic events requiring a response time of less than 5 minutes. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:23-31
| Level | Name | Definition | Response Time |
|---|---|---|---|
| S0 | Catastrophic | Chain halted, assets at risk, or exploits in progress | < 5 min |
| S1 | Critical | Degraded service, potential asset risk, or security vulnerability | < 15 min |
| S2 | Major | Service disruption, no immediate asset risk | < 30 min |
| S3 | Minor | Performance degradation, no service impact | < 2 hours |
Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:25-31
Emergency Response Architecture
The response to a disaster follows a strictly coordinated flow managed by the Incident Commander (IC). The IC holds single-point authority for declaring severity and making go/no-go decisions regarding rollbacks or network pauses. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:70-80
Incident Management Flow
The following diagram illustrates the high-level coordination flow during a detected S0/S1 incident.
The flow ensures that critical incidents are immediately escalated to the IC and Technical Lead for root cause analysis (RCA) and recovery execution. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:94-140, crates/x3-relayer/PHASE_13F_MASTER_INDEX.md:188-250
Core Recovery Playbooks
Chain Halt Recovery
A chain halt is identified when no blocks are produced for more than 5 minutes or finality stalls for over 10 minutes. Recovery involves mandatory validator status checks and potentially wiping corrupted databases or forcing connectivity via specific bootnodes. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:94-110
- Validator Status Check:
curl -H "Content-Type: application/json" -d '{"id":1, "jsonrpc":"2.0", "method": "system_health"}' http://localhost:9944 - Database Wipe (Scenario B):
rm -rf /var/lib/x3-data/chains/*/db/followed by a sync from peers.
Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:118-125, launch-gates/DISASTER_RECOVERY_RUNBOOK.md:155-170
State Corruption and Rollback
State corruption occurs when Merkle roots mismatch or the canonical supply does not equal the sum of all ledgers. This requires an immediate network-wide emergency halt. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:246-260
Bridge Exploit Response
Active exploits on the bridge require a response in under 5 minutes. The primary action is to call the bridge pause extrinsic. If the bridge cannot be paused via extrinsic, the entire network is halted to prevent unauthorized asset draining. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:366-385
Infrastructure Validation and Testing
Disaster recovery readiness is maintained through systematic validation and "War Game" exercises.
Recovery Verification Tasks
Infrastructure must be validated at T-10d before launch, including backups and restore-from-backup tests. Sources: crates/x3-relayer/PHASE_13F_INFRASTRUCTURE_VALIDATION.md:10-25, crates/x3-relayer/PHASE_13F_INFRASTRUCTURE_VALIDATION.md:520-550
| Validation Task | Objective | Target |
|---|---|---|
| Restore Test | Verify backup integrity on staging | RTO < 30 min |
| Database Backup | Ensure daily/hourly encryption | 30-day retention |
| Off-site Storage | Verify backups are not on the same server | S3/Remote verified |
| RPO Check | Measure maximum potential data loss | < 5 min |
Sources: crates/x3-relayer/PHASE_13F_INFRASTRUCTURE_VALIDATION.md:520-565
War Game Scenarios
The team conducts simulations (e.g., "Multiple RPC Providers Down") to identify procedural gaps. Success is measured by detection time (target < 1 min) and full recovery (target < 15 min). Sources: crates/x3-relayer/PHASE_13F_VERIFICATION_EXERCISES.md:25-50
Disaster Recovery Commands
Below are critical commands used during emergency operations:
# Emergency Halt for Validators
sudo systemctl stop x3-validator
# Backup corrupted DB for forensics
tar -czf /backup/corrupted-db-$(date +%s).tar.gz /var/lib/x3-data/chains/
# Export state at specific block for rollback
x3-chain-node export-state \
--chain x3-mainnet-raw.json \
--base-path /var/lib/x3-data \
--block [BLOCK_HASH] > state-export-rollback.json
Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:160-165, launch-gates/DISASTER_RECOVERY_RUNBOOK.md:300-305
Conclusion
The Disaster Recovery system for X3 ATOMIC STAR is built on a foundation of rigorous documentation and pre-launch verification. By establishing clear severity levels, assigned response roles, and automated recovery playbooks for chain stalls, consensus failures, and bridge exploits, the network ensures that even in S0 catastrophic scenarios, asset safety is prioritized and service restoration is choreographed. Sources: launch-gates/DISASTER_RECOVERY_RUNBOOK.md:1-20, launch-gates/PROOF_EXECUTION_GUIDE.md:220-250