LongMemEval & LoCoMo

Relevant source files

The following files were used as context for generating this wiki page:

• BENCHMARKS.md
• paper/drafts/cortex-v2-4-technical-note.md
• paper/reproducer/run_full.sh
• benchmarks/locomo/run-retrieval.ts
• benchmarks/longmemeval/run.ts
• README.md
• benchmarks/locomo/run.ts

LongMemEval and LoCoMo are the primary benchmark suites for evaluating the retrieval performance of the CORTEX memory substrate. These benchmarks test the ability of an AI agent to maintain and retrieve relevant information from long-term conversational haystacks. CORTEX uses these evaluations to validate its neuroscience-grounded architecture, specifically its ability to achieve high recall without the use of LLM-based rerankers.

Sources: paper/drafts/cortex-v2-4-technical-note.md:15-30, BENCHMARKS.md:3-8

Evaluation Architecture

The CORTEX benchmark harness implements a standardized retrieval pipeline. It isolates the retrieval component from answer generation to measure the accuracy of the memory substrate independently.

Benchmark Execution Flow

This diagram illustrates the sequence of operations for evaluating CORTEX against a specific question in the benchmark datasets.

The execution flow ensures that each question is independent by clearing the agent's memory before ingesting the relevant session haystack.
Sources: benchmarks/longmemeval/run.ts:65-115, benchmarks/locomo/run-retrieval.ts:80-130

Hybrid Retrieval Scoring

CORTEX retrieves memories using a linear combination of factors. The system computes a hybrid score for each memory node to determine its relevance to a query.

Factor	Weight	Description
Cosine Similarity	0.45	Semantic embedding match using vector distance
Text Match	0.18	Exact substring hit within the memory content
Recency	0.12	30-day exponential half-life decay
Norm Resonance	0.10	Ebbinghaus stability-adjusted importance score
Priority Boost	0.05	Boost based on manual priority levels (P0-P4)
Emotional Boost	0.10	Boost derived from VAD (Valence-Arousal-Dominance)

Sources: paper/drafts/cortex-v2-4-technical-note.md:85-100, README.md:215-225

LongMemEval Implementation

LongMemEval (ICLR 2025) consists of 500 questions testing five core long-term memory abilities. CORTEX evaluates against two versions: longmemeval_oracle and longmemeval_s (full-haystack).

Core Features

• Independent Ingestion: The harness ingests all conversational sessions associated with a question as separate memories.
• Session Identification: The system converts session turns into single text blocks and maps them to unique sessionId values.
• MRR Scoring: The harness calculates the Mean Reciprocal Rank (MRR) based on where the correct evidence session appears in the top-K retrieved results.

Performance by Question Type

Question Type	Capability Tested
Temporal Reasoning	Reasoning over dates and chronological order
Multi-Session	Connecting information across different conversations
Knowledge Update	Handling facts that change over time
Single-Session	Retrieval within a single conversational context

Sources: benchmarks/longmemeval/run.ts:25-45, BENCHMARKS.md:30-45

LoCoMo Implementation

LoCoMo (ACL 2024) evaluates very long-term conversational memory across 10 conversations containing 1,536 to 1,986 questions.

Categories and Retrieval

The benchmark categorizes questions into four primary types for retrieval scoring:

1. Single-hop: Direct factual retrieval.
2. Multi-hop: Reasoning across multiple steps.
3. Temporal: Handling dates and time-based sequences.
4. Open-domain: Broad inference across the conversational history.

Evidence Tracking

The LoCoMo harness uses regex to extract session numbers from evidence IDs (e.g., "D1:3" maps to session 1). It compares these against retrieved results to determine recall accuracy.

The sequence demonstrates the retrieval-only methodology, which excludes the answer generation phase to focus on the memory substrate's effectiveness.
Sources: benchmarks/locomo/run-retrieval.ts:40-65, benchmarks/locomo/run.ts:25-50

Benchmark Results

CORTEX v2.4 results demonstrate state-of-the-art retrieval without LLM reranking.

Summary Metrics

Metric	LongMemEval (Full-Haystack)	LoCoMo (Overall)
Recall@1	89.4	58.0
Recall@5	97.8	88.6
Recall@10	97.8	93.7
MRR	93.0	70.5

Sources: paper/drafts/cortex-v2-4-technical-note.md:120-135, BENCHMARKS.md:15-25

Reproducibility

The project provides a shell script and a Docker harness to reproduce reported metrics.

Reproducer Components

1. run_full.sh: Executes the entire benchmark chain for both LongMemEval and LoCoMo.
2. longmemeval/run.ts: The TypeScript runner for LongMemEval.
3. locomo/run-retrieval.ts: The TypeScript runner for LoCoMo retrieval scoring.

Environment Requirements

• Node.js: Version 22+ or 25.6.1 (as specified in Appendix A).
• Database: PostgreSQL 16 with pgvector extension.
• Embeddings: Voyage-3 (1024-dim) is the default requirement for baseline results.

Sources: paper/reproducer/run_full.sh:20-45, paper/drafts/cortex-v2-4-technical-note.md:255-270, README.md:270-285

Reproducer Execution Procedure

1. Add VOYAGE_API_KEY and DATABASE_URL to the .env file.
2. Run docker compose up -d db to start the PostgreSQL instance.
3. Execute bash paper/reproducer/run_full.sh to run the full suite.
4. Locate raw JSON artifacts in paper/artifacts/v2-baseline/.

Sources: paper/reproducer/run_full.sh:5-15, paper/reproducer/README.md:10-25

LongMemEval and LoCoMo serve as the critical verification points for CORTEX. By achieving high recall scores without LLM rerankers, CORTEX proves that its hippocampal architecture efficiently separates patterns and manages long-term memory resonance. These benchmarks establish CORTEX as a production-ready substrate for agentic AI applications requiring durable conversational memory.