Core Concepts

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/app/main.go
• internal/embeddings/ollama.go
• internal/embeddings/openai.go
• internal/qdrant/client.go
• internal/search/service.go
• setup-ollama.md

MauQNotes is a semantic search application designed to function as a personal search engine for daily notes. It leverages vector embeddings to understand the semantic meaning of text, allowing users to search based on concepts and questions rather than just keywords. The system ingests text notes, converts them into numerical vector representations using an embedding model (like Ollama or OpenAI), and stores these vectors in a Qdrant vector database.

The core functionality is divided into two main processes: ingestion and querying. Ingestion involves processing and storing notes, while querying involves searching for the most relevant notes based on a user's query. The entire workflow is orchestrated by the search service, which coordinates between the embedding generation module and the Qdrant vector database client.

Sources: cmd/app/main.go:7-29, internal/search/service.go

Architecture Overview

The application follows a simple, modular architecture centered around the search service. The main function acts as the entry point, demonstrating a typical use case: first ingesting a set of notes, and then performing a query against them.

The diagram below illustrates the high-level data flow for both ingestion and querying.

Search Service

The search package is the central coordinator of the application, providing the primary business logic for ingestion and querying.

Sources: internal/search/service.go

Initialization

Before any operation, the service must be initialized using search.Init(). This function reads the EMBEDDING_MODEL environment variable to determine the correct vector dimension size for the Qdrant collection. If the variable is not set, it defaults to nomic-embed-text.

Model Name	Vector Dimension
nomic-embed-text	768 (Default)
all-minilm	384
mxbai-embed-large	512

Note: The system uses a hardcoded map to associate model names with their vector dimensions. If a different model is used, this map must be updated.

Sources: internal/search/service.go:13-32, setup-ollama.md

Ingestion

The IngestNotes function handles the process of adding notes to the vector database.

Process Flow:

1. A new Qdrant client is created.
2. The vector dimension is determined via Init().
3. A Qdrant collection is created if it doesn't already exist, using the determined dimension and Cosine distance metric.
4. Each note string is passed to embeddings.FromOllama to generate a vector embedding.
5. A qdrant.Point is created for each note, containing a unique ID, the generated vector, and a payload. The payload stores the original text and a set of default tags.
6. All created points are sent to Qdrant in a single Upsert operation.

The sequence diagram below visualizes the ingestion process.

Querying

The Query function searches for notes that are semantically similar to a given query string.

Process Flow:

1. A new Qdrant client is created.
2. The query string is passed to embeddings.FromOllama to generate a query vector.
3. The query vector is sent to the Qdrant client's Search method, along with the desired number of results (topK).
4. The search results are returned from Qdrant.
5. A naive re-ranking is applied: if a result's text payload contains the query string (case-insensitive), its score is boosted slightly.
6. The final, potentially re-ranked, results are returned.

The sequence diagram below visualizes the query process.

Embedding Generation

Embeddings are numerical representations of text. This project supports generating embeddings via a local Ollama instance or the remote OpenAI API. The choice is determined by which function is called, though the search service currently defaults to FromOllama.

Sources: internal/embeddings/ollama.go, internal/embeddings/openai.go

Ollama Embeddings

The embeddings.FromOllama function communicates with a local Ollama server.

• Endpoint: It sends a POST request to /api/embeddings on the URL specified by the OLLAMA_URL environment variable, which defaults to http://localhost:11434.
• Request: The request body specifies the model (from EMBEDDING_MODEL env var, defaults to nomic-embed-text) and the text prompt.
• Response: It expects a JSON response containing the vector embedding, which is then converted from []float64 to []float32 for use with the Qdrant client.

Sources: internal/embeddings/ollama.go:13-71, setup-ollama.md

OpenAI Embeddings

The embeddings.FromOpenAI function provides an alternative for generating embeddings using OpenAI's API.

• Endpoint: It sends a POST request to https://api.openai.com/v1/embeddings.
• Authentication: It requires an OPENAI_API_KEY environment variable, which is sent as a Bearer token in the Authorization header.
• Request: The request body specifies the model (from EMBEDDING_MODEL env var) and the input text.

Sources: internal/embeddings/openai.go:15-61

Vector Storage (Qdrant)

The application uses Qdrant as its vector database, managed through a dedicated HTTP client in the qdrant package.

Sources: internal/qdrant/client.go

Qdrant Client

The qdrant.Client is responsible for all communication with the Qdrant database. It is configured via environment variables.

Environment Variable	Description
QDRANT_URL	The base URL of the Qdrant instance (e.g., http://localhost:6333).
QDRANT_API_KEY	The API key for authentication, if required.
QDRANT_COLLECTION	The name of the collection to use for storing points.

The client validates that QDRANT_URL and QDRANT_COLLECTION are set before executing any requests.

Sources: internal/qdrant/client.go:60-79

Core Data Structures

The diagram below shows the main structs used by the Qdrant client for API requests.

Key Operations

• CreateCollectionIfNotExists(vecSize, distance): Sends a PUT request to /collections/{name}. It configures the collection with a specific vector size and a distance metric (Cosine). It gracefully handles cases where the collection already exists (HTTP 409).
  Sources: internal/qdrant/client.go:84-105
• Upsert(points []Point): Sends a PUT request to /collections/{name}/points?wait=true to insert or update a batch of points. The wait=true query parameter ensures the operation is completed before the server responds.
  Sources: internal/qdrant/client.go:107-124
• Search(queryVec []float32, limit int): Sends a POST request to /collections/{name}/points/search. The request body includes the query vector and the number of results to return. It is configured to always include the payload in the results.
  Sources: internal/qdrant/client.go:126-159