CogBench

Relevant source files

The following files were used as context for generating this wiki page:

• benchmarks/cogbench/README.md
• benchmarks/cogbench/scoring.ts
• benchmarks/cogbench/run.ts
• benchmarks/cogbench/generate.ts
• benchmarks/cogbench/types.ts
• benchmarks/cogbench/client.ts
• BENCHMARKS.md

CogBench is a benchmark suite that evaluates the cognitive capabilities of AI memory architectures. It moves beyond simple retrieval accuracy to test how systems manage temporal validity, belief updates, novelty detection, and emotional salience. The suite consists of 49 scenarios and 135 evaluation queries derived from computational neuroscience principles. Sources: benchmarks/cogbench/README.md:1-20, BENCHMARKS.md:91-100

Cognitive Task Architecture

CogBench evaluates seven distinct cognitive capabilities. Each task targets a specific biological or psychological memory function. Sources: benchmarks/cogbench/README.md:22-38, benchmarks/cogbench/types.ts:13-29

Evaluated Capabilities

Task	Scenarios	Cognitive Basis	Description
Temporal Validity	11	Episodic indexing	Enforces time-scoped retrieval and expiry compliance.
Reconsolidation	10	Labile recall window	Updates existing beliefs when new facts arrive.
Novelty Detection	5	CA1 comparator	Distinguishes novel input from redundant information.
Emotional Recall	8	Amygdala modulation	Prioritizes retrieval of emotionally salient memories.
Cross-Agent Transfer	5	Social learning	Transfers knowledge between different agent instances.
Compounding Intelligence	5	Sleep consolidation	Improves retrieval through offline dream cycles.
Procedural Learning	6	Basal ganglia	Acquires skills through repeated execution.

Sources: benchmarks/cogbench/README.md:22-38, BENCHMARKS.md:104-112

Execution Flow

The benchmark process follows a three-step pipeline: dataset generation, scenario execution, and composite scoring. Sources: benchmarks/cogbench/README.md:52-80, benchmarks/cogbench/run.ts:55-120

Benchmarking Workflow

The diagram shows the sequential flow from data generation to final composite scoring. Sources: benchmarks/cogbench/run.ts:55-150, benchmarks/cogbench/client.ts:32-60

Scoring Engine

CogBench uses a deterministic scoring engine to calculate performance across multiple levels. Sources: benchmarks/cogbench/scoring.ts:15-130

Composite Score Calculation

The engine computes a Composite Score using the geometric mean of all per-task mean scores. This method ensures that a score of 0 in any single task results in a total composite score of 0, requiring excellence across all capabilities. Sources: benchmarks/cogbench/scoring.ts:121-127, benchmarks/cogbench/README.md:46-50

Per-Task Metrics

The engine extracts specific metrics for each task to provide granular feedback. Sources: benchmarks/cogbench/scoring.ts:54-99

Task	Primary Metrics
Temporal Validity	Precision, Recall, Expiry Compliance Rate
Reconsolidation	Update Success Rate, Labile Window Compliance
Novelty Detection	Novelty AUC (Area Under Curve), TPR, FPR
Emotional Recall	Emotional Recall Advantage (ERA), Decay Resistance
Compounding Intelligence	Compound Gain, Synapse Utilization

Sources: benchmarks/cogbench/scoring.ts:60-90, benchmarks/cogbench/README.md:40-45

Data Structures

Scenario Definition

A Scenario is the fundamental unit of testing. It contains memory fixtures to be ingested and queries to evaluate the system's response. Sources: benchmarks/cogbench/types.ts:77-88

export interface Scenario {
  id: string;
  taskId: TaskId;
  description: string;
  memories: MemoryFixture[];
  queries: QueryFixture[];
  expected: ExpectedOutcome[];
  config?: Record<string, unknown>;
}

Sources: benchmarks/cogbench/types.ts:77-88

Memory Fixtures

Memory fixtures represent the data ingested into the system. They include metadata such as temporal bounds and emotional charge. Sources: benchmarks/cogbench/types.ts:35-50

• validFrom / validUntil: ISO-8601 timestamps for temporal validity.
• emotionalContext: Valence levels (e.g., positive-high).
• priority: Importance level (0-4).

Sources: benchmarks/cogbench/types.ts:40-49

Benchmark Implementation Details

Extended Client Operations

The CogBench client extends standard CORTEX operations to interface with cognitive subsystems. Sources: benchmarks/cogbench/client.ts:10-25

1. Hippocampal Encoding: The client runs hippocampalEncode to perform Dentate Gyrus pattern separation and CA1 novelty detection during ingestion. Sources: benchmarks/cogbench/client.ts:104-125
2. Valence Analysis: It analyzes the emotional content of memories to set decayResistance and recallBoost. Sources: benchmarks/cogbench/client.ts:128-148
3. Labile Windows: The client manages reconsolidate calls, which are only valid if a memory was recently recalled (within a 1-hour window). Sources: benchmarks/cogbench/client.ts:250-275

Anti-Gaming Measures

To ensure technical accuracy, CogBench implements several anti-gaming protocols:

• Seeded Randomization: Datasets are synthetically generated from templates with seeded randomization. Sources: benchmarks/cogbench/generate.ts:48-60
• Agent Isolation: A fresh, isolated agent is created for every scenario to prevent data leakage. Sources: benchmarks/cogbench/run.ts:100-120
• Deterministic Formulas: All scoring and metrics calculation formulas are deterministic. Sources: benchmarks/cogbench/scoring.ts:15-30

Conclusion

CogBench provides a rigorous framework for validating cognitive memory functions in AI agents. By testing capabilities such as reconsolidation and compounding intelligence, it ensures that memory systems act as living cognitive architectures rather than static vector databases. Sources: benchmarks/cogbench/README.md:12-20, BENCHMARKS.md:91-95