Memory & Context Management
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Memory and Context Management in Clawdbot is a multi-layered system designed to provide agents with continuity, specialized domain knowledge, and persistent state across execution sessions. The system distinguishes between immediate task context, long-term vector-based memory, and persistent agent-specific knowledge. By utilizing a combination of file-system state tracking and vector databases like LanceDB, Clawdbot ensures that agents can maintain a "current understanding" while efficiently handling large volumes of historical data through compaction and selective recall.
The core architecture relies on distinct context layers—Outer Agent State, Persistent Agent Memory, and Task Context—to orient agents within the execution flow. This structure prevents context overwhelming by processing information in a prioritized order: skimming high-level state, reading accumulated memory, and focusing on the immediate task.
Sources: extensions/open-prose/skills/prose/primitives/session.md:16-56, extensions/memory-lancedb/index.ts:12-18
Memory Architecture & Components
The memory system is divided into two primary categories: short-term execution state and long-term vector memory (LTM).
1. Vector Memory (LanceDB)
The memory-lancedb extension provides long-term memory with vector search capabilities using OpenAI embeddings. It features "auto-recall" and "auto-capture" mechanisms to automatically inject relevant memories and store new information.
Sources: extensions/memory-lancedb/index.ts:7-11
Data Structure (MemoryEntry):
| Field | Type | Description |
|---|---|---|
id | string | Unique UUID for the entry |
text | string | The raw text information |
vector | number[] | Embedding vector for search |
importance | number | Score between 0-1 |
category | string | Category (preference, decision, fact, entity, other) |
createdAt | number | Timestamp of storage |
| Sources: extensions/memory-lancedb/index.ts:24-31, extensions/memory-lancedb/index.ts:153-157 |
2. File-System State Management
For OpenProse programs, state is persisted in a .prose/ directory. This includes run-specific bindings, execution traces in state.md, and persistent agent memory files.
Sources: extensions/open-prose/skills/prose/state/filesystem.md:12-45
Memory Data Flow
The following diagram illustrates how user input triggers memory recall and how outcomes are captured back into long-term storage.
The diagram shows the lifecycle of memory during an agent interaction, from initial embedding-based search to post-execution storage.
Sources: extensions/memory-lancedb/index.ts:316-398
Context Layering
Agents receive context from multiple sources, which must be processed in a specific layering order to maintain efficiency.
Context Layers
- Outer Agent State: Context from the orchestrating VM (e.g., current phase, prior steps).
- Persistent Agent Memory: Accumulated knowledge from previous segments stored in
memory.md. - Task Context: Specific input for the current session (code to review, plans to evaluate).
- Execution Scope: Information about block invocations (ID, depth, parent ID).
Sources: extensions/open-prose/skills/prose/primitives/session.md:18-80
Processing Order
To avoid context bloat, agents are instructed to:
- Skim outer state to find their place in the big picture.
- Read their memory to recall what they already know.
- Focus on task context for the immediate work.
- Synthesize prior knowledge with the current task.
Sources: extensions/open-prose/skills/prose/primitives/session.md:52-56
Memory Compaction and Persistence
Persistent agents maintain state via memory.md files. This requires compaction, which is distinct from summarization.
Compaction Principles
Compaction focuses on preserving specifics while dropping reasoning chains.
- Preserve: Specific file locations (e.g.,
src/auth/login.ts:67), exact findings (e.g., "SQL injection risk"), and numbers (e.g., "73% coverage"). - Drop: Intermediate reasoning, false starts, and verbose quotes.
Sources: extensions/open-prose/skills/prose/primitives/session.md:118-160
This sequence shows the requirement for persistent agents to build upon existing memory and record the current session's impact.
Sources: extensions/open-prose/skills/prose/primitives/session.md:275-310
Memory Tools & CLI
Management of memory is handled through specialized tools and CLI commands.
Agent Tools
| Tool Name | Action | Parameters |
|---|---|---|
memory_recall | Search LTM | query, limit |
memory_store | Save to LTM | text, importance, category |
memory_forget | Delete memory | query or memoryId |
| Sources: extensions/memory-lancedb/index.ts:192-310 |
CLI Management
The clawdbot memory command provides status, indexing, and search capabilities.
- Status:
clawdbot memory status --agent <id> --deep(checks provider, model, sources, and vector/FTS status). - Indexing:
clawdbot memory index --force(synchronizes file-system memory files into the search manager). - Search:
clawdbot memory search "<query>"(performs manual vector search for debugging).
Sources: src/cli/memory-cli.ts:251-447
Memory Source Labels
The CLI tracks memory across two main types of sources:
- Memory Files:
MEMORY.mdandmemory/*.mdin the workspace. - Session Transcripts:
.jsonlfiles in the agent-specific sessions directory.
Sources: src/cli/memory-cli.ts:54-65
System Prompt Integration
The system-prompt.ts module dynamically constructs instructions for memory usage based on the runtime environment. If memory_search or memory_get tools are available, the agent is instructed to run these tools before answering anything regarding prior work or decisions.
Sources: src/agents/system-prompt.ts:40-50
Summary
Clawdbot's Memory & Context Management system ensures that agents are not merely stateless LLM instances. By leveraging a structured file system for execution state and a vector database for long-term recall, the project enables complex, multi-step workflows. Persistent agents take active responsibility for their continuity through rigorous memory compaction, while the CLI and system prompts provide the necessary infrastructure to keep these memories searchable and relevant.