Obsidian Vault Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Obsidian Vault Integration allows Nyansa to transform a local folder of Markdown notes into a searchable semantic memory. This system mounts the vault into the Docker environment, parses its structure, and indexes the content for Retrieval-Augmented Generation (RAG).
Sources: README.md:17-21, services/memory-api/README.md:1-5
Architecture and Data Flow
The integration relies on Docker bind mounts to share the host's Obsidian vault with internal services. The memory-api service acts as the primary orchestrator for ingestion and search.
The diagram above shows the data flow from the host filesystem to the vector database.
Sources: README.md:46-95, docker-compose.yml:138-140, docker-compose.yml:257-259
Service Access Modes
| Service | Mount Path | Access Mode | Purpose |
|---|---|---|---|
nyansa-memory-api | /vault | ro (Read-only) | Parsing and indexing for semantic search. |
nyansa-n8n | /data/obsidian-vault | rw (Read/Write) | Workflow-based note manipulation. |
Sources: docker-compose.yml:138-140, docker-compose.yml:257-259, services/memory-api/README.md:83
Vault Ingestion Pipeline
The ingestion process converts raw Markdown files into searchable vector points. It uses incremental indexing to minimize processing time by only re-embedding changed notes.
1. File Discovery and Reading
The listNotes function scans the vault directory for .md files. It ignores hidden directories (e.g., .obsidian, .trash) and specific non-note folders like node_modules. Files are read and decoded, defaulting to windows-1252 if UTF-8 decoding fails.
Sources: services/memory-api/src/vault/vault.ts:31-62, services/memory-api/src/vault/vault.ts:69-76
2. Note Parsing
The parseNote function extracts structured data from the Markdown content:
- Frontmatter: Extracts YAML metadata using
splitFrontmatter. - Title: Resolves the title from the
titlefrontmatter key, the first H1 heading, or the filename. - Tags: Collects and normalizes tags from frontmatter and inline
#tagsyntax. - Links: Identifies
[[wikilinks]]and![[embeds]].
Sources: services/memory-api/src/markdown/parse.ts:12-42, services/memory-api/src/markdown/parse.ts:48-60
3. Chunking Strategy
The chunkNote logic splits notes into smaller segments to fit embedding model constraints.
- Heading Awareness: Splits occur at ATX headings.
- Heading Paths: Chunks retain context through paths like
Setup > Docker. - Size Limits: Paragraphs are packed up to
CHUNK_MAX_CHARS(default 1500). - Splitting: Oversized blocks are split by lines, then sentences, then characters.
Sources: services/memory-api/README.md:57-61, services/memory-api/src/markdown/chunk.ts
4. Embedding and Storage
Each chunk is prefixed with its note title and section path before being sent to the nomic-embed-text model. The resulting vectors are stored in Qdrant as points.
Configuration and Environment
The integration is configured via environment variables defined in the .env file and utilized in docker-compose.yml.
| Variable | Description | Default |
|---|---|---|
OBSIDIAN_VAULT_PATH | Absolute path to the vault on the host machine. | Required |
EMBEDDING_MODEL | The Ollama model used for generating vectors. | nomic-embed-text |
CHUNK_MAX_CHARS | Maximum character limit per indexed chunk. | 1500 |
MEMORY_API_TOKEN | Bearer token required for /ingest and /search. | Required |
Sources: docker-compose.yml:243-254, services/memory-api/README.md:83, README.md:144-154
Incremental Indexing Logic
Nyansa avoids unnecessary re-indexing by comparing the current vault state against stored metadata in Qdrant.
- SHA-256 Hashing: The system hashes the raw bytes of every note during scanning.
- Index Versioning: A version string is generated from the
CHUNKER_VERSION, the embedding model name, andCHUNK_MAX_CHARS. - Planning: The
planIngestlogic identifies:- New/Changed: Notes with a new hash or different index version.
- Deleted: Notes present in Qdrant but missing from the filesystem.
- Unchanged: Notes where both hash and index version match.
Sources: services/memory-api/src/ingest/ingest.ts:76-88, services/memory-api/src/vault/vault.ts:78-80, services/memory-api/src/ingest/ingest.ts:58-62
Summary
The Obsidian Vault Integration provides the foundation for Nyansa's "second brain" capabilities. By combining efficient filesystem scanning, intelligent Markdown parsing, and incremental vector storage, it ensures that local notes are always available for AI-powered semantic search and retrieval.