Incident Management & Auto-Recovery
Relevant source files
The following files were used as context for generating this wiki page:
The Incident Management & Auto-Recovery system provides autonomous detection, escalation, and resolution protocols for the NeXifyAI platform. It operates as a self-healing runtime that monitors system health, detects drift, and repairs failures within established governance bounds. Every decision, error, and recovery action persists in the platform's Brain (Qdrant) to enable continuous learning and improvement.
Sources: ARCHITECTURE.md, README.md
System Architecture
The recovery architecture relies on four pillars to maintain operational stability:
- Governance Bootstrap: Validates system health before allowing execution.
- Circuit Breakers: Prevents cascading failures across services.
- Dead Letter Queues (DLQ): Captures failed operations for later replay or analysis.
- Brain Persistence: Feeds every error and recovery insight into the knowledge base.
Sources: ARCHITECTURE.md
Operational Flow
The following diagram illustrates the flow from system startup through drift detection to recovery or halt.
The system prioritizes auto-repair but halts execution if governance bounds are exceeded to prevent unsafe states.
Sources: ARCHITECTURE.md
Incident Classification and Escalation
Incidents are classified by priority levels (P0 to P3), each triggering specific response times and actor requirements.
| Priority | Definition | Timeout | Action |
|---|---|---|---|
| P0 | Critical Blocker / System Failure | 15 Minutes | Immediate notification to CEO and all stakeholders |
| P1 | High Impact / Repeated Failure | 60 Minutes | Alert Domain Expert; CEO escalation after 3 failures |
| P2 | Medium Impact | 4 Hours | Route to Review Agent and Project Manager |
| P3 | Low Impact / Scheduled | 24 Hours | Process in next business day batch |
Sources: services/api/agents/brain_agents/order-workflow-specialist.md:38-43, services/api/agents/brain_agents/senior-quality-auditor.md:144-149
Escalation Matrix
The system employs a 4-level escalation matrix based on agent performance and system health.
Automated Recovery Patterns
The platform implements standardized recovery patterns for common failures identified during runtime.
- Stuck Orders: Triggers a timeout, followed by escalation and reassignment to a new agent.
- Agent Unavailability: Executes two retries before initiating the expert matching algorithm to find a replacement.
- Dead Dependencies: Signals all dependent orders and triggers a CEO-level review.
- Quality Gate Failure: Routes the task back to a
REWORKstate with specific feedback for the agent. - Brain Unavailability: Falls back to keyword-based routing and queues all pending Brain writes for later processing.
Sources: services/api/agents/brain_agents/order-workflow-specialist.md:94-100
Monitoring and Health Audits
Autonomous agents continuously audit the system to detect gaps before they result in P0 incidents.
Monitoring Specialist
The Monitoring Specialist tracks the "4 Golden Signals": Latency, Traffic, Errors, and Saturation. It uses Uptime Kuma for heartbeats and polls the health endpoints of Qdrant (port 6333) and FastAPI (port 8001).
Sources: services/api/agents/brain_agents/monitoring-specialist.md:1-25
Inventory Brain Scanner
This agent cross-references the Brain state with the live system. It flags discrepancies in:
- Agent Health: Verifies profile length and mandatory blocks.
- Brain Integrity: Audits staleness and low-confidence vectors.
- Infrastructure: Monitors container health and SSL certificate expiry.
- API Coverage: Tracks auth requirements and endpoint health.
Sources: services/api/agents/brain_agents/inventory-brain-scanner.md:15-55
Post-Incident Protocol
After every P0 or P1 incident, the Senior Quality Auditor conducts a mandatory postmortem within one hour of resolution. This process identifies root causes and prevents recurrence.
Postmortem Template
Postmortems must include:
- Impact Analysis: Systems affected and data loss assessment.
- Root Cause: 5 Whys analysis.
- Gate Failure Analysis: Determining which quality gates failed to catch the issue.
- Prevention Plan: Specific code or architecture changes.
- Brain Enrichment: Lessons stored in the knowledge base for future recovery cycles.
Sources: services/api/agents/brain_agents/senior-quality-auditor.md:88-100, services/api/agents/brain_agents/senior-quality-auditor.md:157
Recovery Audit Flow
Conclusion
The Incident Management & Auto-Recovery system ensures that NeXifyAI remains "Operational by Design." By combining proactive inventory scanning, rigid quality gates, and autonomous escalation matrices, the platform minimizes downtime and ensures that every failure results in a stronger, more resilient knowledge base.
Sources: README.md, ARCHITECTURE.md