Vector Storage with Qdrant

Relevant source files

The following files were used as context for generating this wiki page:

• docker-compose.yml
• services/memory-api/src/clients/qdrant.ts
• services/memory-api/src/ingest/ingest.ts
• services/memory-api/src/domain.ts
• services/memory-api/README.md
• README.md

Qdrant serves as the vector database for the Nyansa project, providing high-performance storage and semantic search capabilities for embedded Obsidian vault notes. It stores document chunks as high-dimensional vectors alongside rich metadata, enabling the memory-api service to perform Retrieval-Augmented Generation (RAG).

The system integrates Qdrant through the nyansa-qdrant Docker service, which is isolated within the internal nyansa network. It requires an API key for all operations except health checks, ensuring secure communication with the memory-api and n8n services.
Sources: README.md:28-30, docker-compose.yml:144-158, services/memory-api/README.md:1-5

Architecture and Integration

Qdrant operates as a stateful service within the Docker Compose stack. It stores its data in a named volume to ensure persistence across container restarts. The memory-api interacts with Qdrant through the QdrantStore client implementation of the VectorStore interface.

Service Configuration

Parameter	Source	Description
QDRANT_API_KEY	.env	Required security token for API access.
QDRANT_URL	Internal	http://nyansa-qdrant:6333
Collection Name	MEMORY_API	Default: obsidian_notes
Storage Path	Volume	/qdrant/storage inside container.

Sources: docker-compose.yml:153-157, services/memory-api/README.md:95-101

Data Schema and Payload

Qdrant stores document data in a single collection. Each "point" in the collection represents a note chunk and consists of a 128-bit UUID, a vector (embedding), and a JSON payload.

Payload Structure

The ChunkPayload contains metadata used for filtering and citation.
Sources: services/memory-api/src/domain.ts:40-55

Field	Type	Description
path	string	Vault-relative POSIX path.
title	string	Note title (from H1 or filename).
tags	string[]	Deduplicated, lower-cased tags.
section	string	Heading path (e.g., Setup > Docker).
text	string	The raw Markdown content of the chunk.
note_hash	string	SHA-256 of the raw file for incremental indexing.
index_version	string	Version hash of models and chunking logic.

Indexing Fields

The QdrantStore class explicitly creates keyword and integer indexes for efficient filtering during search.

• Keyword Indexes: path, tags
• Integer Index: chunk_index

Sources: services/memory-api/src/clients/qdrant.ts:31-35

Interaction Logic

The QdrantStore class manages the lifecycle of vector data, including collection initialization, incremental state tracking, and semantic search.

Collection Initialization

Before ingestion, the service ensures the collection exists and verifies vector dimensionality. If the existing collection's vector size differs from the current embedding model (e.g., nomic-embed-text), it throws an UpstreamError to prevent corrupted data.
Sources: services/memory-api/src/clients/qdrant.ts:50-71

Incremental Indexing Strategy

Incremental indexing avoids re-embedding unchanged notes. The system stores the note_hash and index_version in the payload of the first chunk (chunk_index: 0) of every note.

Deterministic Point IDs

Point IDs are generated as deterministic RFC 4122 UUIDs derived from the file path and chunkIndex. This ensures that re-indexing a note automatically overwrites its previous points in Qdrant.
Sources: services/memory-api/src/ingest/ingest.ts:168-174

Search Implementation

Semantic search is performed via the POST /search endpoint. The QdrantStore translates application-level search options into Qdrant Query API calls.

Search Flow

1. Vectorization: memory-api converts the text query into a vector via Ollama.
2. Filtering: If tags are provided, a Qdrant must filter is applied to the tags keyword field.
3. Thresholding: A score_threshold (Cosine distance) filters out irrelevant results.
4. Retrieval: Qdrant returns the top N hits with full payloads.

// Example search logic in QdrantStore
const body: Record<string, unknown> = { query: vector, limit: options.limit, with_payload: true };
if (options.minScore !== undefined) {
  body['score_threshold'] = options.minScore;
}
if (options.tags !== undefined && options.tags.length > 0) {
  body['filter'] = { must: [{ key: 'tags', match: { any: options.tags } }] };
}

Sources: services/memory-api/src/clients/qdrant.ts:107-124

Summary

The Qdrant implementation provides a resilient and secure vector storage layer for Nyansa. By utilizing deterministic IDs and storing metadata directly within vector payloads, the system maintains a lightweight architecture that keeps the search index in sync with a local Obsidian vault without requiring an external relational database for state management.
Sources: services/memory-api/README.md:75-80, README.md:14-25