Crash Recovery & Resilience
Relevant source files
The following files were used as context for generating this wiki page:
The Crash Recovery & Resilience system in Cogmo ensures that the agentic loop remains durable and consistent across failures, process restarts, and unexpected LLM behaviors. By leveraging event-driven orchestration and durable execution, the system can resume operations from the last known good state without duplicating side effects like double-posting messages or re-charging LLM credits.
This framework is built upon Inngest to provide exactly-once semantics for critical operations while allowing non-critical UI updates to re-execute. It addresses a taxonomy of failures ranging from hardware crashes to agent logic loops.
Sources: CLAUDE.md, design/crash-recovery.md, design/agent-resilience.md
Durable Execution Architecture
Cogmo uses a "durability map" within the handle-message orchestrator to define how different code blocks behave during a retry or recovery after a crash.
Exactly-Once Operations
Critical operations that interact with the database or external LLMs are wrapped in step.run() to ensure they only execute once. This includes:
- Database Upserts: Inserting user or assistant messages into the
messagestable. - LLM Calls: Summarization calls during message compaction and tool execution.
- Tool Side Effects: Any tool that performs an external action (e.g., sending an email or creating a file).
Re-executing Blocks
Certain logic is designed to re-execute on every retry to ensure the agent has the most up-to-date context:
- Token Counting: Recalculated to ensure threshold decisions are based on current data.
- Memory Retrieval: Re-running
memory.recallensures the agent sees facts that might have been updated by concurrent processes.
Sources: design/crash-recovery.md, changelog.d/2026-04-11-crash-recovery-contract-summarization-llm.md
Flow of Durable Message Handling
The following diagram illustrates the flow of a message through the durable orchestrator, highlighting the boundary between durable steps and re-executing logic.
The diagram shows how step.run creates memoization checkpoints for DB inserts and LLM summarization while context loading remains dynamic.
Sources: design/crash-recovery.md, changelog.d/2026-04-11-crash-recovery-contract-summarization-llm.md
Failure Taxonomy & Repair
Resilience is not just about surviving crashes but also about recovering from logic failures within the agent's "thought" process. Cogmo categorizes these into specific failure modes.
Failure Categories
| Category | Description | Mitigation Strategy |
|---|---|---|
| Crash | Process death, OOM, or hardware failure. | Inngest retry + step.run memoization. |
| Tool Error | Tool returns an error or malformed JSON. | Error injected back into LLM context for self-correction. |
| Stuck Loop | Agent calls same tool with same args repeatedly. | Loop detection logic triggers a "degraded reply" or manual intervention. |
| Budget Exhaustion | LLM exceeds token or cost limits. | Immediate termination with a summary of partial work. |
Sources: design/agent-resilience.md, CLAUDE.md
In-Loop Repair
When a non-fatal error occurs (such as a malformed tool response), the system uses an in-loop repair mechanism. Instead of failing the entire message handler, the error is treated as a "virtual" tool result and fed back to the agent.
Implementation Details
Reconcile on Failure
The system includes specialized logic for reconciling state when a recovery occurs mid-operation. The reconcile-on-failure.ts module handles edge cases where a process might have died after an external action but before the DB could be updated.
Key functions:
reconcileToolEffect(): Checks external systems to see if a requested action was actually completed before retrying.handleStuckLoop(): Monitors the tool invocation history for identical repeated calls.
Sources: src/agent/reconcile-on-failure.ts, src/agent/repair.ts
Testing Resilience
Resilience is verified using "Replay Regression Tests." These tests use the @inngest/test framework to simulate a partially completed execution and verify that:
- Memoized steps (like
summarize-prefix) are not called a second time. - The streaming output section re-runs to ensure the user receives the final response even if the first attempt crashed mid-stream.
Sources: design/testing.md, changelog.d/2026-04-11-crash-recovery-contract-summarization-llm.md
Summary
The Crash Recovery & Resilience system provides a safety net for the Cogmo runtime. By distinguishing between durable side effects and dynamic context re-execution, it ensures the system remains reliable at a personal scale. Through the use of Inngest for durability and internal repair loops for agent logic, Cogmo can survive both infrastructure failures and "hallucinated" tool calls.