Developer Guide

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/app/main.go
• internal/search/service.go
• internal/qdrant/client.go
• internal/embeddings/ollama.go
• internal/embeddings/openai.go
• setup-ollama.md

MauQNotes is a personal semantic search engine for notes. It is designed to take a collection of text notes, convert them into numerical representations (embeddings), and store them in a vector database for fast and context-aware retrieval. The application is written in Go and integrates with an embedding model provider (like Ollama or OpenAI) and the Qdrant vector database.

The core functionality demonstrated in the application is a two-step process: first, ingesting a list of notes into the Qdrant database, and second, performing a semantic search query against those notes to find the most relevant results. The system is configured via environment variables, allowing flexibility in choosing the embedding model and connecting to the Qdrant instance.

Sources: cmd/app/main.go:7-29, internal/search/service.go:11-15

System Architecture

The application is composed of three main internal packages orchestrated by the main application entrypoint: search, embeddings, and qdrant. The main function initiates the two primary workflows: ingestion and querying.

This diagram shows the high-level interaction between the application's components.
Sources: cmd/app/main.go, internal/search/service.go, internal/embeddings/ollama.go, internal/qdrant/client.go

Configuration

The application is configured using environment variables, which are loaded from a .env file.

Variable	Description	Default Value	Source File
EMBEDDING_MODEL	The name of the embedding model to use.	nomic-embed-text	internal/search/service.go:12-15
QDRANT_URL	The base URL for the Qdrant instance.	(none)	internal/qdrant/client.go:70-72
QDRANT_API_KEY	The API key for authenticating with Qdrant.	(none)	internal/qdrant/client.go:71-73
QDRANT_COLLECTION	The name of the collection in Qdrant to use.	(none)	internal/qdrant/client.go:72-74
OLLAMA_URL	The URL for the Ollama API service.	http://localhost:11434	internal/embeddings/ollama.go:23-26
OPENAI_API_KEY	The API key for the OpenAI service.	(none)	internal/embeddings/openai.go:26-30

Sources: internal/search/service.go:11, internal/qdrant/client.go:68, internal/embeddings/ollama.go:21, internal/embeddings/openai.go:25

Core Workflows

The application has two primary functions: ingesting notes and querying them.

Note Ingestion

The ingestion process converts text notes into vector embeddings and stores them in the Qdrant database.

This sequence diagram illustrates the flow of data and function calls during the note ingestion process.

Sources: cmd/app/main.go:16-19, internal/search/service.go:39-65, internal/qdrant/client.go:108-143

Semantic Querying

The querying process converts a text query into a vector and uses it to find the most similar notes in the Qdrant database.

This sequence diagram shows the steps involved in executing a semantic search query.

Sources: cmd/app/main.go:22-25, internal/search/service.go:67-87, internal/qdrant/client.go:145-182

Component Details

search Service

This service acts as the central orchestrator.

• Init(): Reads the EMBEDDING_MODEL environment variable and returns the corresponding vector dimension. This is crucial for creating the Qdrant collection with the correct size.
• IngestNotes(notes []string): Manages the entire ingestion pipeline. It creates the Qdrant collection, iterates through notes to generate embeddings via the embeddings service, constructs qdrant.Point objects, and calls the qdrant client to upsert them.
• Query(topK int, q string): Manages the search pipeline. It generates an embedding for the query string, calls the qdrant client's search method, and performs a simple reranking by slightly boosting the score of results that contain the query string as a substring.

Sources: internal/search/service.go:10-88

embeddings Service

This package is responsible for converting text into vector embeddings by calling an external API. The application currently uses FromOllama by default in the search service.

Ollama Provider

The FromOllama function sends a POST request to the /api/embeddings endpoint of a running Ollama instance. The URL defaults to http://localhost:11434 but can be overridden by the OLLAMA_URL environment variable.

• Request Body (ollamaRequest):
  • model: The name of the model (e.g., nomic-embed-text).
  • prompt: The text to be embedded.
• Response (ollamaResponse):
  • embedding: An array of float64 values representing the vector.

The function converts the returned []float64 to []float32 to match the type required by the qdrant client.

Sources: internal/embeddings/ollama.go:17-70, internal/search/service.go:50

OpenAI Provider

An alternative provider, FromOpenAI, is available but not used by the search service. It sends a POST request to https://api.openai.com/v1/embeddings and requires OPENAI_API_KEY and EMBEDDING_MODEL to be set.

Sources: internal/embeddings/openai.go:24-69

Supported Models & Dimensions

The vector dimension is hardcoded based on the model name.

Model Name	Dimension	Recommended by
nomic-embed-text	768	setup-ollama.md, search/service.go
all-minilm	384	setup-ollama.md, search/service.go
mxbai-embed-large	512	search/service.go

Note: setup-ollama.md lists mxbai-embed-large as having 1024 dimensions, while internal/search/service.go implements it with 512 dimensions. The code in search/service.go is the effective configuration.

Sources: internal/search/service.go:19-29, setup-ollama.md:19-22

qdrant Client

This client handles all HTTP communication with the Qdrant vector database.

This diagram shows the structure of the Qdrant client and the Point data structure it uses for upserting data.

Sources: internal/qdrant/client.go:22-76

Key methods and their corresponding API calls:

• CreateCollectionIfNotExists: Sends a PUT request to /collections/{collection_name}. It ignores a 409 Conflict status code, making the operation idempotent.
• Upsert: Sends a PUT request to /collections/{collection_name}/points?wait=true with a batch of points.
• Search: Sends a POST request to /collections/{collection_name}/points/search with the query vector and a limit.

The Point struct is central to data ingestion:

Field	Type	Description
ID	interface{}	A unique identifier for the point. An integer is used in the app.
Vector	[]float32	The embedding vector for the note.
Payload	map[string]any	Arbitrary JSON data associated with the vector. The app stores the original text and tags.

Sources: internal/qdrant/client.go:22-30, internal/search/service.go:52-58

Local Setup (Ollama)

To run the application locally, an Ollama instance with an embedding model is required.

1. Install Ollama: Download from https://ollama.ai/download.
2. Pull an Embedding Model: The recommended model is nomic-embed-text.

   ollama pull nomic-embed-text
   

3. Configure Environment: Ensure your .env file has the correct EMBEDDING_MODEL and QDRANT_* variables set.
4. Run Application: The main function provides a simple example of ingesting and querying notes.

Sources: setup-ollama.md:3-12, cmd/app/main.go:7-29