Markdown Parsing Engine

Relevant source files

The following files were used as context for generating this wiki page:

• services/memory-api/src/markdown/parse.ts
• services/memory-api/src/markdown/chunk.ts
• services/memory-api/src/markdown/frontmatter.ts
• services/memory-api/src/markdown/lines.ts
• services/memory-api/src/domain.ts
• services/memory-api/src/ingest/ingest.ts

The Markdown Parsing Engine extracts structured data and semantic chunks from Obsidian notes for use in the Nyansa RAG (Retrieval-Augmented Generation) pipeline. It transforms raw Markdown text into a ParsedNote object containing metadata and subsequently into multiple Chunk objects optimized for vector embedding.

This engine handles frontmatter extraction, wiki-link resolution, tag collection, and heading-aware content splitting. It ensures that semantic context, such as heading hierarchies, is preserved within each chunk to improve search relevance.

Sources: services/memory-api/src/markdown/parse.ts:1-12, services/memory-api/src/domain.ts:1-35, services/memory-api/README.md

Core Data Structures

The engine operates on specific TypeScript interfaces that define the lifecycle of a note from raw file to indexed chunk.

ParsedNote

This structure holds the initial result of the parsing phase before the note is split into chunks.

Field	Type	Description
path	string	Vault-relative POSIX path (e.g., projects/nyansa.md).
title	string	Resolved title from frontmatter, H1, or filename.
frontmatter	Record<string, unknown>	Parsed YAML metadata.
tags	string[]	Lower-cased, deduplicated, and sorted tags.
links	WikiLink[]	List of internal wiki-links and embeds.
body	string	Content remaining after frontmatter removal.

Sources: services/memory-api/src/domain.ts:17-30

Chunk

The chunk represents the unit of text sent to the embedding model.

Field	Type	Description
index	number	Zero-based position within the parent note.
section	string	Flattened heading path (e.g., Setup > Docker).
headings	string[]	Array of parent headings.
text	string	The actual content segment.

Sources: services/memory-api/src/domain.ts:32-38

Parsing Logic and Workflow

The parsing process follows a linear transformation from raw string to structured object.

The parsing workflow prioritizes frontmatter extraction before processing the body text.
Sources: services/memory-api/src/markdown/parse.ts:12-38

Frontmatter Extraction

The engine uses splitFrontmatter to identify a YAML block at the start of the file. The block must be delimited by --- and can be closed by either --- or .... If the YAML is invalid, the engine reports an error but continues processing the body.

Sources: services/memory-api/src/markdown/frontmatter.ts:12-32

Title Resolution

The engine resolves the note title using the following priority:

1. The title field in the YAML frontmatter.
2. The first Level 1 Heading (# Title) found in the body.
3. The filename (excluding the .md extension) derived from the path.

Sources: services/memory-api/src/markdown/parse.ts:41-47

Tag and Link Collection

Tags are collected from two sources: the tags or tag fields in the frontmatter, and inline #tags within the prose. Links are extracted from Obsidian-style [[WikiLinks]] and ![[Embeds]].

• Masking: Inline code spans (surrounded by backticks) are masked with spaces before scanning to prevent code snippets from being misidentified as tags or links.
• Normalization: Tags are lower-cased and deduplicated.
• WikiLinks: Parsed into target, heading, and alias components.

Sources: services/memory-api/src/markdown/parse.ts:50-104, services/memory-api/src/markdown/lines.ts:40-42

Chunking Strategy

The chunkNote function splits the body into segments suitable for vector stores based on heading structures and character limits.

The chunking strategy ensures that context is not lost during fragmentation.
Sources: services/memory-api/src/markdown/chunk.ts:27-46

Heading-Aware Splitting

The engine tracks the "heading path" as it scans the document. When a new heading is encountered, the current section is closed, and a new one starts. Deeper headings are appended to the path, while shallower headings reset the path to their level.

Sources: services/memory-api/src/markdown/chunk.ts:48-89

Text Constraints and Overflow

The engine respects a maxChars configuration (defaulting to 1500).

• Block Packing: Paragraphs are combined into a single chunk until the maxChars limit is reached.
• Code Preservation: Fenced code blocks are kept whole unless the block itself exceeds maxChars.
• Oversized Splitting: If a single block is too large, it is split sequentially by:
  1. Lines (newline characters).
  2. Sentences (punctuation followed by whitespace).
  3. Characters (hard cut at maxChars).

Sources: services/memory-api/src/markdown/chunk.ts:91-137

Line and Code Processing

The scanLines utility serves as the low-level scanner for both the parser and the chunker. It identifies lines that belong to fenced code blocks to ensure they are excluded from tag extraction and heading identification.

)
InCode --> InCode: Line in Code
InCode --> Prose: Detected Closing Fence (```)

The scanner uses regex to detect ATX headings and code fences.
Sources: [services/memory-api/src/markdown/lines.ts:1-38](https://github.com/Geobatpo07/nyansa/blob/b6101033e572cc656f7842ce9da33590c4cb0897/services/memory-api/src/markdown/lines.ts#L1-L38)

### Key Regular Expressions
The engine relies on specific regex patterns for Markdown syntax:
- **Inline Tags:** `(?:^|[\s(])#([\p{L}\p{N}_/-]+)`
- **WikiLinks:** `(!?)\[\[([^[\]\n]+?)\]\]`
- **ATX Headings:** `^ {0,3}(#{1,6})[ \t]+(.*?)(?:[ \t]+#+)?[ \t]*$`
- **Code Fences:** `^ {0,3}(`{3,}|~{3,})`

Sources: [services/memory-api/src/markdown/parse.ts:7-9](https://github.com/Geobatpo07/nyansa/blob/b6101033e572cc656f7842ce9da33590c4cb0897/services/memory-api/src/markdown/parse.ts#L7-L9), [services/memory-api/src/markdown/lines.ts:8-9](https://github.com/Geobatpo07/nyansa/blob/b6101033e572cc656f7842ce9da33590c4cb0897/services/memory-api/src/markdown/lines.ts#L8-L9)

## Integration with Ingestion

During the ingestion process, the `Ingestor` uses the parsing engine to prepare data for the `Embedder`.

1. **Hashing:** The raw file is hashed using SHA-256 for incremental indexing logic.
2. **Contextual Embedding:** Before embedding, the chunk text is prefixed with its title and section path: `${title} > ${section}\n\n${text}`.
3. **Point Mapping:** Each chunk is assigned a deterministic UUID (point ID) based on the file path and chunk index.

Sources: [services/memory-api/src/ingest/ingest.ts:122-166](https://github.com/Geobatpo07/nyansa/blob/b6101033e572cc656f7842ce9da33590c4cb0897/services/memory-api/src/ingest/ingest.ts#L122-L166)

The Markdown Parsing Engine ensures that the semantic structure of Obsidian notes is accurately reflected in the Nyansa vector store, enabling precise and context-aware search results.