Quick Start Guide

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/app/main.go
• setup-ollama.md
• internal/search/service.go
• internal/qdrant/client.go
• internal/embeddings/ollama.go
• internal/embeddings/openai.go

MauQNotes is a personal semantic search engine designed to index and query notes. It leverages an embedding model to convert text into numerical vectors and a vector database (Qdrant) to store and search these vectors efficiently. This allows for finding notes based on their semantic meaning rather than just keyword matching. The core functionality demonstrated in the application is the ability to ingest a list of notes and then perform a similarity search against them with a natural language query.

The system is primarily built around Go, using Ollama for local embedding generation and Qdrant for vector storage. The main application orchestrates the two primary processes: ingesting notes by converting them to vectors and upserting them into Qdrant, and querying by converting a search term into a vector to find the most relevant notes.

Sources: cmd/app/main.go:7-29, internal/search/service.go:42-83

1. Setup and Configuration

Before running the application, you must set up the required services: Ollama for embeddings and Qdrant for the vector database.

1.1. Ollama Setup

Ollama is used to generate text embeddings locally.

1. Install Ollama: Download and install from the official website: https://ollama.ai/download.
2. Pull Embedding Model: After installation, pull the recommended embedding model. The default model is nomic-embed-text.

   ollama pull nomic-embed-text
   

3. Verify Ollama: Ensure the Ollama service is running, typically at http://localhost:11434.

Sources: setup-ollama.md:3-13, internal/embeddings/ollama.go:23-26

1.2. Environment Variables

The application is configured using a .env file. The following variables are required:

Variable	Description	Default Value
QDRANT_URL	The base URL for the Qdrant instance.	(None)
QDRANT_API_KEY	The API key for authenticating with Qdrant (optional).	(None)
QDRANT_COLLECTION	The name of the collection in Qdrant to store the notes.	(None)
EMBEDDING_MODEL	The name of the embedding model to use.	nomic-embed-text
OLLAMA_URL	The URL for the Ollama API service.	http://localhost:11434
OPENAI_API_KEY	The API key for OpenAI, required if using an OpenAI embedding model.	(None)

Sources: internal/qdrant/client.go:70-76, internal/search/service.go:13-16, internal/embeddings/ollama.go:20-31, internal/embeddings/openai.go:27-31

1.3. Embedding Models

The application supports several embedding models, each with a different vector dimension size. The model choice affects the Qdrant collection configuration.

Model Name	Vector Dimension	Notes
nomic-embed-text	768	Recommended, default model.
all-minilm	384	Smaller and faster.
mxbai-embed-large	512	Larger and potentially more accurate.

The application automatically determines the dimension based on the EMBEDDING_MODEL environment variable.

Sources: setup-ollama.md:21-25, internal/search/service.go:18-33

2. System Architecture

The application consists of several interconnected components that handle the logic for embedding, storage, and search.

The diagram below shows the high-level architecture of the MauQNotes system.

• main.go: The entry point of the application, orchestrating the ingestion and query flows.
• search.Service: Contains the core business logic for ingesting and querying notes.
• embeddings.Service: An interface for generating vector embeddings from text. The primary implementation uses Ollama (FromOllama), but an OpenAI implementation (FromOpenAI) also exists.
• qdrant.Client: A client for interacting with the Qdrant vector database API, handling collection creation, point upserting, and searching.
• Ollama API: The external service that provides text-to-vector embedding generation.
• Qdrant DB: The external vector database that stores and indexes the note vectors for fast retrieval.

Sources: cmd/app/main.go, internal/search/service.go, internal/qdrant/client.go, internal/embeddings/ollama.go

3. Core Processes

The application has two main functions: ingesting notes into the vector database and querying the database to find relevant notes.

3.1. Ingestion Flow

The ingestion process takes a list of text notes, converts each one into a vector embedding, and stores it in the Qdrant database.

This sequence diagram illustrates the note ingestion process.

1. main.go calls search.IngestNotes with a slice of strings.
2. The search service initializes, determining the required vector dimension from the configured EMBEDDING_MODEL.
3. It ensures a collection exists in Qdrant with the correct dimension and distance metric (Cosine).
4. For each note, it calls embeddings.FromOllama to generate a vector.
5. The note text, its vector, and some metadata are packaged into a qdrant.Point.
6. Finally, all points are sent to Qdrant via the client.Upsert method.

Sources: cmd/app/main.go:15-18, internal/search/service.go:42-69, internal/embeddings/ollama.go:18-70, internal/qdrant/client.go:138-163

3.2. Query Flow

The query process takes a user's search string, converts it into a vector, and uses that vector to perform a similarity search in Qdrant.

This sequence diagram illustrates the search query process.

1. main.go calls search.Query with a search string and the desired number of results (topK).
2. The search string is converted into a query vector using embeddings.FromOllama.
3. The qdrant.Client sends this vector to the Qdrant search endpoint.
4. Qdrant returns a list of the most similar points based on cosine similarity.
5. A naive re-ranking step is applied: if the original note text contains the query string, its score is slightly boosted.
6. The final results are returned to the main application for display.

Sources: cmd/app/main.go:21-23, internal/search/service.go:71-83, internal/embeddings/ollama.go:18-70, internal/qdrant/client.go:165-202

4. Key Components and Data Structures

4.1. Qdrant Client

The qdrant.Client encapsulates all communication with the Qdrant database.

This class diagram shows the structure of the Qdrant client.

• New(): Constructor that creates a new client, loading configuration from environment variables.
• CreateCollectionIfNotExists(): Sends a PUT request to /collections/{name} to create a new vector collection if it doesn't already exist. It handles a 409 Conflict status gracefully.
• Upsert(): Sends a PUT request to /collections/{name}/points to add or update a batch of points.
• Search(): Sends a POST request to /collections/{name}/points/search to find vectors similar to the query vector.

Sources: internal/qdrant/client.go:67-202

4.2. Data Structures

The application uses several structs to model data for API requests to Qdrant and Ollama.

Struct Name	File	Purpose
Point	internal/qdrant/client.go	Represents a single data point in Qdrant, containing an ID, a vector, and a JSON payload.
upsertRequest	internal/qdrant/client.go	The request body for batch-upserting points to Qdrant.
searchRequest	internal/qdrant/client.go	The request body for performing a search query in Qdrant.
ollamaRequest	internal/embeddings/ollama.go	The request body sent to the Ollama /api/embeddings endpoint.
openAIRequest	internal/embeddings/openai.go	The request body sent to the OpenAI /v1/embeddings endpoint.

Sources: internal/qdrant/client.go:26-55, internal/embeddings/ollama.go:13-17, internal/embeddings/openai.go:13-17

A qdrant.Point is the fundamental unit of data stored in the vector database.

// From internal/qdrant/client.go:26-31
type Point struct {
	ID      interface{}    `json:"id"`
	Vector  []float32      `json:"vector"`
	Payload map[string]any `json:"payload,omitempty"`
}

• ID: A unique identifier for the point.
• Vector: The numerical representation of the note text.
• Payload: Arbitrary JSON data associated with the vector, such as the original text and tags.

Sources: internal/qdrant/client.go:26-31, internal/search/service.go:56-62