Incident Response & Kill-Switch
Relevant source files
The following files were used as context for generating this wiki page:
The Incident Response and Kill-Switch systems provide a governed, multi-stage framework for identifying, containing, and resolving technical and security anomalies within the Isabella project. These systems ensure that high-severity events, such as data breaches or financial loss, trigger automated containment actions while requiring human oversight for final resolution. The framework operates under a "Zero Trust" architecture, where all critical recovery actions require hardware-backed authentication and dual control.
Sources: legal/INCIDENT-RESPONSE-POLICY.md:1-24, governance/README.md:1-24, scripts/authz/isabella_scopes_catalog_v2.json:344-360
Incident Classification and Severities
The project categorizes incidents into four severity levels based on the impact on secrets, financial integrity, and service availability. An automated classifier evaluates incident summaries to assign the appropriate level.
Severity Tiers
| Severity | Criteria | Response Time |
|---|---|---|
SEV-1 | Exposure of secrets, authentication bypass, or financial loss. | Immediate |
SEV-2 | Tenant isolation failure, ledger integrity issues, or data corruption. | < 4 hours |
SEV-3 | Service degradation or non-critical policy errors. | < 24 hours |
SEV-4 | Visual defects or documentation errors. | < 1 week |
Sources: legal/INCIDENT-RESPONSE-POLICY.md:3-9, src/lib/governance/incident.ts:20-25
Classification Logic
The classifyIncident function utilizes regular expressions to map summary keywords to severity levels. Keywords related to "tenant", "isolation", or "ledger" trigger SEV-2, while "secret", "bypass", or "financial" trigger SEV-1.
Sources: src/lib/governance/incident.ts:37-43
Incident Response Runbook
The response process follows a strictly sequential 8-phase runbook to ensure evidence preservation and systematic remediation.
The runbook ensures that evidence is preserved with checksum-verified logs before credentials are revoked or root causes are addressed.
Sources: legal/INCIDENT-RESPONSE-POLICY.md:12-21, src/lib/governance/incident.ts:27-35
Kill-Switch Mechanism
The Kill-Switch is a critical emergency containment module activated by SEV-1 or SEV-2 incidents. It transitions the system from a normal state to restricted states, such as egress-frozen, to halt further damage.
Activation and Execution
When activated via the /api/v1/kill-switch/activate endpoint, the system generates a unique event ID and starts a 10-step automated sequence. These steps include actions like FREEZE_EGRESS and eventual RESUME after manual intervention.
Sources: tests/kill-switch.test.ts:16-32, server.ts:1071-1078
State Transitions
The system requires an administrator to execute steps sequentially. Certain steps are flagged as humanRequired, meaning the system will not proceed without explicit approval.
Sources: tests/kill-switch.test.ts:34-70, legal/INCIDENT-RESPONSE-POLICY.md:23-26
API Endpoints
| Endpoint | Method | Role | Description |
|---|---|---|---|
/api/v1/kill-switch/status | GET | Authenticated | Returns current state and remaining steps. |
/api/v1/kill-switch/activate | POST | Admin | Triggers the emergency state. |
/api/v1/kill-switch/:eventId/step | POST | Admin | Executes the next automated action in the sequence. |
/api/v1/kill-switch/:eventId/resolve | POST | Admin | Restores system to normal state. |
Sources: server.ts:1067-1094
Quantum Mesh Recovery
The Quantum Mesh subsystem includes specific recovery procedures for hardware and provider failures. The RecoveryIncident data structure tracks affected components and automated actions taken.
Recovery Procedures
- Worker Failure: If a worker hangs, the system kills the process, marks the job as retryable, and spawns a replacement.
- HSM Unavailable: If the Primary Hardware Security Module fails, the system blocks new signatures and allows only the reading of previously signed results.
- BookPI/Postgres Down: If the audit chain persistence fails, the system stops high-impact executions (QPU/remote) and maintains an encrypted local journal.
Sources: src/lib/quantum/recovery.ts:10-108
Recovery Governance Workflow
Recovery operations for critical events like HSM failure require the recovery:activate scope, which demands hardware-backed identity and dual control.
Sources: src/lib/quantum/recovery.ts:68-80, scripts/authz/isabella_scopes_catalog_v2.json:344-360
Conclusion
The Incident Response and Kill-Switch modules serve as the project's primary defense against catastrophic failure. By combining automated detection and containment with mandatory human-in-the-loop validation for restoration, the system maintains operational integrity even during severe security breaches. All emergency actions are recorded in an immutable audit trail, ensuring post-incident accountability and root-cause analysis.
Sources: governance/README.md:18-24, src/lib/isabella-inference-engine.ts:32-60