RAG Ingestion Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• services/memory-api/src/ingest/ingest.ts
• services/memory-api/src/ingest/plan.ts
• services/memory-api/src/markdown/parse.ts
• services/memory-api/src/markdown/chunk.ts
• services/memory-api/src/domain.ts
• services/memory-api/README.md

The RAG (Retrieval-Augmented Generation) Ingestion Pipeline transforms raw Markdown files from an Obsidian vault into searchable vector embeddings. This system manages the lifecycle of vault data by parsing content, splitting it into semantic chunks, generating high-dimensional vectors via Ollama, and persisting the results in a Qdrant vector store.

Sources: services/memory-api/README.md:3-8, services/memory-api/src/ingest/ingest.ts:77-84

Pipeline Architecture

The pipeline operates as a centralized service within the memory-api module. It coordinates file system scanning, content analysis, and communication with external AI and database services.

Core Ingestion Logic

The Ingestor class manages the execution flow and ensures only one ingestion process runs at a time. It uses a versioning system (indexVersion) to trigger re-indexing if chunking parameters or the embedding model change.

Ingestion Modes

The pipeline supports two operational modes:

• Incremental: Processes only notes with changed content hashes or outdated index versions.
• Full: Forces re-indexing of every note in the vault.

Sources: services/memory-api/src/ingest/ingest.ts:57-69, services/memory-api/src/ingest/plan.ts:16-29

Error Handling and Resilience

The system tracks consecutive failures during interactions with Ollama or Qdrant. If the MAX_CONSECUTIVE_UPSTREAM_FAILURES threshold (default: 3) is reached, the pipeline aborts the run to prevent excessive error logs. Notes that fail or are skipped during an aborted run are retried in subsequent executions.

Sources: services/memory-api/src/ingest/ingest.ts:12-14, services/memory-api/src/ingest/ingest.ts:119-126

Processing Stages

1. Markdown Parsing

The parser extracts structured data from Obsidian notes, including:

• Frontmatter: YAML metadata blocks.
• Title: Resolved from frontmatter title, the first H1 heading, or the file name.
• Tags: Normalized, lower-cased tags from both frontmatter and inline #tag mentions.
• Links: [[wikilinks]] and ![[embeds]].

Sources: services/memory-api/src/markdown/parse.ts:14-38, services/memory-api/src/domain.ts:16-27

2. Semantic Chunking

The chunkNote function splits the note body into sections based on ATX headings. It maintains a "heading path" (e.g., Setup > Docker) to provide context for each chunk. Paragraphs are packed into chunks up to the CHUNK_MAX_CHARS limit (default: 1500). Code blocks are kept whole unless they exceed the limit, in which case they are split by lines.

Sources: services/memory-api/src/markdown/chunk.ts:25-50, services/memory-api/README.md:48-49

3. Embedding Generation

Each chunk is prefixed with its note title and section path to preserve context before being sent to the embedding model. The pipeline uses the nomic-embed-text model via Ollama. It applies specific prefixes (search_document:) required by the model for document indexing.

Sources: services/memory-api/src/ingest/ingest.ts:184-187, services/memory-api/README.md:50-52

4. Vector Storage

Vectors are stored in Qdrant with a payload containing the original text and metadata.

• Point IDs: Deterministic UUIDs generated from the file path and chunk index. This allows re-indexing to overwrite existing data.
• Payload Fields: Includes path, tags, links, section, note_hash, and index_version.

Sources: services/memory-api/src/ingest/ingest.ts:193-215, services/memory-api/src/domain.ts:40-54

Data Structures

Ingest Report

The pipeline returns a summary of the ingestion run.

Field	Type	Description
mode	IngestMode	Either full or incremental.
notes.scanned	number	Total notes found in the vault.
notes.indexed	number	Number of notes successfully embedded.
notes.deleted	number	Number of notes removed from index.
chunks.upserted	number	Total vector points written to Qdrant.
aborted	string	Reason for early termination, if applicable.

Sources: services/memory-api/src/ingest/ingest.ts:17-31

Chunk Payload Schema

The following table describes the metadata stored with each vector point in Qdrant.

Property	Type	Usage
path	string	Vault-relative path for file reference.
section	string	The heading hierarchy for citations.
note_hash	string	SHA-256 hash used for change detection.
index_version	string	Versioning for model/chunking changes.
tags	string[]	Lower-cased tags for search filtering.

Sources: services/memory-api/src/domain.ts:40-54

Implementation Detail: Deterministic UUIDs

The pipeline ensures that re-indexing a note correctly updates existing entries without creating duplicates. It generates a UUID by hashing the file path and the chunk index.

export function pointId(path: string, chunkIndex: number): string {
  const hex = sha256(`${path}\u0000${chunkIndex}`).slice(0, 32).split('');
  hex[12] = '5'; // version nibble
  hex[16] = ((Number.parseInt(hex[16] ?? '0', 16) & 0x3) | 0x8).toString(16); // RFC 4122 variant
  const h = hex.join('');
  return `${h.slice(0, 8)}-${h.slice(8, 12)}-${h.slice(12, 16)}-${h.slice(16, 20)}-${h.slice(20, 32)}`;
}

Sources: services/memory-api/src/ingest/ingest.ts:221-228

Summary

The RAG Ingestion Pipeline provides an incremental, fault-tolerant mechanism to sync Obsidian notes with a vector store. By combining heading-aware chunking with deterministic ID generation, it maintains an efficient and context-rich index suitable for semantic search and AI retrieval.