Chaos Engineering & Testing
Relevant source files
The following files were used as context for generating this wiki page:
Chaos Engineering within the X3 project is a systematic approach to validating system resilience by injecting adversarial conditions into the RC5 validators and relayer infrastructure. The primary goal is to simulate production failures—such as network partitions, resource starvation, and process crashes—to prove that the X3 Atomic Star architecture can maintain consensus and recover automatically without human intervention. This suite ensures that the system satisfies "Go/No-Go" readiness criteria for mainnet deployment.
The testing framework is structured around several specialized toolsets, including the Chaos Harness, Attack Vector scripts, and the Resilience Orchestrator. These tools are designed to run alongside active burn-in tests (like the RC5 72-hour test) to monitor real-time health metrics, invariant violations, and recovery timelines. Sources: CHAOS_TESTING_GUIDE.md:3-10, launch-gates/comprehensive-mainnet-readiness.sh:267-275
Chaos Testing Architecture
The chaos engineering architecture relies on a multi-phase execution strategy that progresses from baseline inventory audits to full-scale fault injection. It is integrated into the project's CI/CD pipeline and launch-gate verification process.
Resilience Components
The suite is comprised of three primary orchestration layers:
- Chaos Harness (
rc5_chaos_harness.sh): A systematic framework that provides multiple attack vectors for resource and infrastructure stress testing. Sources: CHAOS_TESTING_GUIDE.md:12-14 - Attack Vectors (
rc5_attack_vectors.sh): A script focused on targeted exploit testing, including RPC overflows and finality race conditions. Sources: CHAOS_TESTING_GUIDE.md:27-28 - Resilience Orchestrator (
rc5_resilience_orchestrator.sh): A master coordinator used to run comprehensive multi-phase testing scenarios ranging from "Light" to "Extreme." Sources: CHAOS_TESTING_GUIDE.md:43-45
Chaos Execution Flow
The following diagram illustrates how chaos is injected and monitored within the X3 environment:
The diagram shows the interaction between the Chaos Control scripts and the target Validator systems, resulting in an automated assessment report. Sources: CHAOS_TESTING_GUIDE.md:14-25, CHAOS_TESTING_GUIDE.md:126-135
Chaos Testing Scenarios and Intensity
Testing is categorized by intensity levels and specific scenarios to allow for progressive stress testing without immediate catastrophic failure.
Intensity Scaling
The framework utilizes a 1-10 intensity scale to define the severity of injected faults:
- Levels 1-4: Minimal to light stress; considered safe for baseline testing.
- Levels 5-8: Moderate to heavy stress; likely to cause temporary unresponsiveness or validator crashes.
- Levels 9-10: Maximum chaos; system may become unresponsive and require manual recovery.
Sources: CHAOS_TESTING_GUIDE.md:104-113
Test Scenario Matrix
| Scenario | Duration | Intensity | Focus Area |
|---|---|---|---|
| Light | ~10 min | 1-2 | Safe baseline, basic network latency. |
| Medium | ~20 min | 3-5 | Moderate stress, process restarts, recovery testing. |
| Heavy | ~30 min | 6-8 | Aggressive adversarial conditions, resource starvation. |
| Extreme | Variable | 9-10 | Maximum stress, requires manual confirmation. |
| Full | ~90 min | 1-8 | Comprehensive multi-phase testing. |
Sources: CHAOS_TESTING_GUIDE.md:47-53, CHAOS_TESTING_GUIDE.md:162-178
Attack Vectors and Fault Injection
The system identifies 10 specific attack vectors that are systematically tested to ensure the protocol's robustness.
Key Attack Vectors
- RPC Numeric Overflow: Testing parameter parsing for block hashes.
- Finality Race Condition: Detection of finality regression.
- Settlement Invariants: Violating state machine logic to check for rollback success.
- Cascade Failure: Simultaneous network, process, memory, and disk chaos.
- Network Partition: Simulating timeouts and packet loss using
tcandiptables.
Sources: CHAOS_TESTING_GUIDE.md:30-41, CHAOS_TESTING_GUIDE.md:86-90
Fault Injection Sequence
This sequence demonstrates the lifecycle of a fault injection from the harness through the network layer to the node, concluding with recovery and monitoring. Sources: CHAOS_TESTING_GUIDE.md:12-25, CHAOS_TESTING_GUIDE.md:76-80, CHAOS_TESTING_GUIDE.md:140-145
Mainnet Readiness Gates
Chaos testing is a mandatory component of the Mainnet Readiness Report. Failure to survive chaos scenarios is an automatic "No-Go" for deployment.
Required Chaos Proofs
- Fault-Injection Tests: Proof of timeout, refund, and replay rejection.
- Determinism Proofs: IXL rollback determinism under stress.
- Symmetry Table: Proving write/read consistency during resource starvation.
- Auto-Heal Verification: Documenting specific incidents where the system self-recovered (e.g., AH-001).
Sources: YOLO_FINISHER_v5.prompt.md:58-62, launch-gates/comprehensive-mainnet-readiness.sh:267-285
Final Assessment Metrics
| Metric | Target | Fail Condition |
|---|---|---|
| Recovery Time | < 30s (Medium Chaos) | > 60s (Heavy Chaos) |
| Finality Monotonicity | Continuous Advancement | Finality Regression Detected |
| Consensus Stability | All validators recover | State Divergence Detected |
| Memory Growth | Stable/Managed | Potential Memory Leak (> 75MB) |
Sources: CHAOS_TESTING_GUIDE.md:140-149, CHAOS_TESTING_GUIDE.md:162-178
Conclusion
Chaos Engineering and Testing at X3 provide the final hardening required for a production-ready environment. By simulating extreme conditions through the Chaos Harness and Resilience Orchestrator, the system validates its ability to maintain data integrity and consensus. Successful completion of the "Extreme" chaos scenarios, with zero detected panics or regressions, marks the final transition to mainnet readiness. Sources: CHAOS_TESTING_GUIDE.md:195-200, launch-gates/comprehensive-mainnet-readiness.sh:305-315