Long-Term Memory System

Relevant source files

The following files were used as context for generating this wiki page:

• design/memory.md
• src/agent/memory-tools.ts
• src/agent/evolution/memory-extraction-schema.ts
• design/architecture.md
• CLAUDE.md

The Long-Term Memory (LTM) system in Cogmo provides a persistent, semantic storage layer that allows the agent to learn and recall facts, preferences, and behavioral patterns across different conversations. Built upon the Hindsight memory server, it utilizes a vector-based approach combined with keyword and graph-based retrieval to ensure high relevance in search results. The system is designed to be "stateless per invocation," where the intelligence resides in the LLM sessions while the memory serves as a specialized store for persistent data.

Sources: design/architecture.md:3-8, design/memory.md:3-12

System Architecture

Cogmo's memory architecture follows a client-server model. The main application interacts with a self-hosted Hindsight server via an HTTP client. Hindsight manages its own PostgreSQL database with pgvector for storage and performs its own LLM-based extraction and reflection.

Component Overview

The following diagram illustrates the high-level interaction between the Cogmo Agent and the Memory System.

Key Components

Component	Responsibility	Source
Hindsight Server	Handles embedding, vector search, deduplication, and consolidation.	design/memory.md:12
MemoryProvider	Interface abstraction for memory access (Recall, Reflect, Retain).	design/memory.md:30-33
Observer	Post-conversation process that extracts facts from transcripts and drains pending memories.	design/memory.md:158-165
Pending Memories	A staging table for live memory_retain calls to ensure uniform classification.	design/memory.md:133-145

The Four Memory Networks

Cogmo categorizes all stored information into four distinct "networks" using Hindsight tags. This allows the agent to distinguish between external objective truths and personal user preferences.

Network	Tag	Description	Examples
World	network:world	External facts about tools, systems, people, and places.	"homelab IP is 10.0.10.10"
Bank	network:bank	Personal facts, preferences, habits, and biographical details.	"allergic to peanuts", "prefers tea"
Opinion	network:opinion	Agent's internal assessments of what works for the user.	"user gets frustrated with verbose info"
Observation	network:observation	Behavioral patterns noticed by the agent over time.	"usually asks about homelab on weekends"

Sources: design/memory.md:149-155, src/agent/evolution/memory-extraction-schema.ts:145-156

Data Flow: Storage and Retrieval

The Staging Write Path

To ensure high-quality classification and prevent "remember to remember" failures during active chat, the system uses a Staging Path for real-time writes.

1. Tool Call: The agent invokes the memory_retain tool.
2. Staging: The content is inserted into the pending_memories table.
3. Idle Trigger: When the conversation goes idle, the Observer wakes up.
4. Classification: The Observer uses a specialized LLM prompt to assign a Network, Compartment, and Trust Tier to the memory.
5. Persistence: The finalized, tagged memory is sent to the Hindsight server's retain() endpoint.

Sources: design/memory.md:133-145, src/agent/memory-tools.ts:35-51

Retrieval Logic

Retrieval is handled via two primary mechanisms: Auto-Recall and Explicit Tools.

• Auto-Recall: Before an agent starts a turn, the system searches memory based on the user's message and injects it as context. This is gated by a heuristic to avoid unnecessary costs on trivial messages (e.g., "Hi").
• Memory Recall Tool: Used for semantic search queries. It searches all networks by default.
• Memory Reflect Tool: Drives an agentic reasoning loop within Hindsight to synthesize answers for complex, multi-hop questions.

Sources: design/memory.md:175-181, src/agent/memory-tools.ts:7-33, src/agent/memory-tools.ts:53-90

Security and Access Control

Memory access is governed by two dimensions of tags: Compartments and Trust Tiers.

Access Dimensions

• Compartments: Lateral isolation of domains (e.g., work, personal, health). Profiles are scoped to specific compartments.
• Trust Tiers: Vertical boundaries defining who can access information. first-party is reserved for user-controlled profiles, while any is safe for third-party extensions.

Core Compartments

The system defines a set of curated core compartments that are always available:

• personal: General life, habits, family.
• work: Professional commitments, projects.
• health: Medical, fitness, dietary needs.
• financial: Money, accounts, taxes.
• technical: Infrastructure, code, system configs.
• misc: Facts fitting no other bucket.

Sources: src/agent/evolution/memory-extraction-schema.ts:30-43, src/agent/evolution/memory-extraction-schema.ts:158-176

Technical Implementation Details

Tool Definitions

Memory interactions are exposed to the agent via Zod-validated tools.

// From src/agent/memory-tools.ts
export const memoryRecall = defineTool({
  name: "memory_recall",
  description: "Search long-term memory for facts...",
  schema: z.object({
    query: z.string().describe("Semantic search query")
  }),
  handler: async (input, service) => {
    const result = await service.memory.recall(input.query);
    // ... returns mapped memories
  }
});

Sources: src/agent/memory-tools.ts:7-27

Classification Schema

The Observer uses the MemoryExtractionSchema to structure its output during the post-conversation extraction phase.

// From src/agent/evolution/memory-extraction-schema.ts
export const ExtractedMemorySchema = z.object({
  fact: z.string().trim().min(1),
  network: MemoryNetworkSchema,
  compartment: MemoryCompartmentSchema,
  trust: MemoryTrustSchema,
  context: z.string().optional()
});

Sources: src/agent/evolution/memory-extraction-schema.ts:98-110

Summary

The Cogmo Long-Term Memory system provides a robust framework for agentic learning by separating real-time interaction from heavy-duty memory processing. By using the Hindsight server as a backend and a staged write path through an Observer, the system ensures that every fact is accurately categorized into networks and compartments, maintaining both high retrieval relevance and strict data isolation.