Developing Memory API

Relevant source files

The following files were used as context for generating this wiki page:

• services/memory-api/package.json
• services/memory-api/src/app.ts
• services/memory-api/src/config.ts
• services/memory-api/src/domain.ts
• services/memory-api/src/ingest/ingest.ts
• services/memory-api/src/markdown/parse.ts
• services/memory-api/src/markdown/chunk.ts
• services/memory-api/README.md
• README.md

Memory API is a TypeScript service that transforms an Obsidian vault into a searchable vector database. It provides Retrieval-Augmented Generation (RAG) capabilities by parsing Markdown notes, splitting them into manageable chunks, generating embeddings via Ollama, and storing them in Qdrant.

The service runs within the Nyansa Docker network and communicates primarily with nyansa-ollama for inference and nyansa-qdrant for storage. It enables incremental indexing to minimize processing time by only re-embedding modified notes.

Sources: services/memory-api/README.md, README.md:19-25

System Architecture

The Memory API follows a port-and-adapter architecture to isolate domain logic from external services like Ollama and Qdrant. The core logic resides in Ingestor and SearchService modules.

High-Level Data Flow

The following diagram illustrates the ingestion pipeline from the Obsidian vault to the vector store:

The ingestion process uses SHA-256 hashes to detect file changes and avoid redundant embedding calls.

Sources: services/memory-api/README.md, services/memory-api/src/ingest/ingest.ts:74-100

Core Components

1. Markdown Parsing

The parseNote function extracts structured data from raw Markdown. It handles YAML frontmatter, ATX headings, tags (both frontmatter and inline #tags), and [[wikilinks]].

• Frontmatter: Extracts titles and tags using the splitFrontmatter utility.
• Headings: Identifies H1 tags to use as fallback titles.
• Tags: Normalizes and deduplicates tags, converting them to lowercase and removing the # prefix.
• Links: Collects internal vault links and media embeds.

Sources: services/memory-api/src/markdown/parse.ts:12-42, services/memory-api/src/markdown/frontmatter.ts:13-25

2. Note Chunking

The chunkNote function splits note bodies into sections based on heading levels. It ensures that chunks do not exceed the CHUNK_MAX_CHARS limit (default 1500).

Heading Paths*: Sections maintain their hierarchy (e.g., Setup > Docker).

• Oversized Blocks: If a paragraph or code block exceeds the limit, the system splits it by lines, then sentences, then characters.
• Empty Notes: Notes without text are indexed using their title to remain searchable.

Sources: services/memory-api/src/markdown/chunk.ts:24-45, services/memory-api/src/domain.ts:29-35

3. Ingestion Management

The Ingestor class coordinates the update cycle. It prevents concurrent runs and handles upstream failures by aborting after three consecutive errors from Ollama or Qdrant.

Sources: services/memory-api/src/ingest/ingest.ts:46-72

API Interface

The service uses Fastify to expose endpoints. All routes except /health require a Bearer token.

Endpoints

Method	Path	Description	Payload Example
GET	/health	Checks status of Ollama and Qdrant.	N/A
POST	/ingest	Triggers vault indexing. Supports full or incremental modes.	{"mode": "incremental"}
POST	/search	Performs semantic search across indexed chunks.	{"query": "setup", "limit": 5}

Sources: services/memory-api/src/app.ts:88-112, services/memory-api/README.md

Search Response Structure

The search result includes the similarity score, the source file path, and the specific section heading.

Sources: services/memory-api/src/domain.ts:71-79

Configuration

Configuration is managed via environment variables and validated using Zod.

Variable	Default	Description
MEMORY_API_TOKEN	(Required)	Bearer token for API authentication (32+ chars).
VAULT_PATH	/vault	Path where the Obsidian vault is mounted.
OLLAMA_URL	http://nyansa-ollama:11434	URL for the local Ollama instance.
QDRANT_URL	http://nyansa-qdrant:6333	URL for the Qdrant vector store.
CHUNK_MAX_CHARS	1500	Maximum character length for a single chunk.

Sources: services/memory-api/src/config.ts:3-15

Development and Testing

The project uses vitest for testing. The test suite avoids external network calls by using in-memory fakes for Ollama and Qdrant.

Build Scripts

• Build: npm run build (compiles TypeScript to dist/).
• Test: npm test (runs the Vitest suite).
• Typecheck: npm run typecheck (runs tsc without emitting files).

Sources: services/memory-api/package.json:11-17

The Memory API provides the critical link between private Markdown notes and AI workflows, ensuring that local LLMs can access specific, relevant context from the user's "second brain."