Ingestion Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Ingestion Pipeline transforms raw content into structured memory nodes within the CORTEX cognitive architecture. It processes text through a sequence of operations including chunking, embedding, entity extraction, and hippocampal encoding to facilitate long-term memory storage and retrieval.
This system supports multiple input vectors including a REST API, a Model Context Protocol (MCP) server, and a file watcher that monitors the local workspace for changes.
Sources: README.md:144-158, src/api/ingest.ts:16-20, src/watcher.ts:13-17
Pipeline Architecture
The pipeline follows a linear execution flow to convert raw text into enriched memory nodes stored in PostgreSQL with pgvector.
Data Flow
The diagram illustrates the sequential stages of the ingestion process from initial input to the final formation of associative connections.
Sources: README.md:144-158, src/ingestion/ingest-markdown.ts:72-148
Core Processing Stages
Chunking and Embedding
The pipeline uses chunkText to break large documents into manageable segments. Each chunk is then converted into a vector representation using embedTexts.
Sources: src/ingestion/ingest-markdown.ts:98-106, src/api/ingest.ts:48-51
Entity and Tag Extraction
The system extracts proper nouns and semantic tags from each text chunk. These metadata attributes facilitate hybrid search and cognitive clustering.
- Entities: Proper nouns identified via regex or LLM extraction.
- Semantic Tags: Categories such as "technical", "financial", or "urgent" based on content keywords.
Sources: src/ingestion/ingest-markdown.ts:102-103, src/api/ingest.ts:54-55
Hippocampal Encoding
CORTEX implements a bio-inspired encoding layer that manages novelty and pattern separation.
- DG (Dentate Gyrus): Performs pattern separation via sparse codes (4096-dim expansion).
- CA1: Detects novelty by comparing incoming patterns against existing memory.
- Surprise Gating: Adjusts the memory's priority and resonance score based on the detected novelty.
Sources: README.md:162-166, src/api/ingest.ts:58-59, src/ingestion/ingest-markdown.ts:110-111
Emotional Valence Analysis
Every memory node receives a six-dimensional emotional vector. This analysis determines the salience of the memory, affecting its decay resistance and recall boost.
| Dimension | Description |
|---|---|
| Valence | Positivity or negativity of the content |
| Arousal | Level of physiological or psychological intensity |
| Dominance | Degree of control or influence described |
| Certainty | How factual or confirmed the information is |
| Relevance | Connection to current agent goals |
| Urgency | Requirement for immediate attention or action |
Sources: src/ingestion/ingest-markdown.ts:140-154, README.md:177-180
Input Sources
File Ingestion
The pipeline supports bulk and individual ingestion of Markdown and Telegram files. It infers priority based on file names and locations.
| Priority | File Patterns / Locations |
|---|---|
| P0 (Critical) | MEMORY.md, STANDING-ORDERS.md, SOUL.md, AGENTS.md |
| P1 (High) | USER.md, YYYY-MM-DD.md (daily logs) |
| P2 (Normal) | enhancement files, /logs/ directory |
| P3 (Low) | /telegram/, /limitless/ contexts |
Sources: src/ingestion/ingest-markdown.ts:25-45, src/ingestion/ingest-markdown.ts:173-195
MCP Server Tools
The Model Context Protocol server exposes ingestion capabilities as tools for AI agents.
cortex_ingest: Stores raw text content into memory.cortex_ingest_file: Ingests a specific file from the disk.cortex_ingest_corpus: Performs a one-time bulk ingestion of the entire V1 corpus.
Sources: src/mcp/server.ts:321-447
File Watcher
watcher.ts monitors the workspace for file additions or changes. It implements a 5-minute debounce to prevent redundant processing during active writing sessions.
Sources: src/watcher.ts:10-75
Database Storage
Finalized memory nodes are stored in the memory_nodes table. The pipeline also populates related tables:
hippocampal_codes: Stores the sparse representation and novelty scores.emotional_valence: Stores the 6D affect vectors and salience metrics.memory_synapses: Stores associative links created byformSynapses.
Sources: src/ingestion/ingest-markdown.ts:113-156, src/api/ingest.ts:61-94
The Ingestion Pipeline ensures that every piece of information entering CORTEX is not merely stored as a vector, but is integrated into a weighted, emotional, and associative cognitive network.