Setting up Qdrant
Relevant source files
The following files were used as context for generating this wiki page:The MauQNotes project utilizes Qdrant as its vector database to store and retrieve notes based on semantic similarity. The internal/qdrant package provides a dedicated HTTP client to interact with a Qdrant instance, abstracting the API calls for creating collections, upserting data points, and performing vector searches. This setup is central to the application's core functionality of building a personal semantic search engine for notes.
The overall process involves generating vector embeddings from notes (using an embedding model like one from Ollama), and then using the Qdrant client to manage this data. The search service orchestrates these operations, from initializing the database settings to ingesting notes and handling user queries.
Sources: internal/qdrant/client.go, internal/search/service.go, cmd/app/main.go
Configuration
The Qdrant client is configured using environment variables. These variables must be present in a .env file or set in the environment for the application to connect to the Qdrant instance correctly.
| Environment Variable | Description | Default Value | Source File |
|---|---|---|---|
QDRANT_URL | The base URL of the Qdrant instance. | (none) | internal/qdrant/client.go:76 |
QDRANT_API_KEY | The API key for authenticating with the Qdrant instance (optional). | (none) | internal/qdrant/client.go:77 |
QDRANT_COLLECTION | The name of the collection to use for storing note vectors. | (none) | internal/qdrant/client.go:78 |
EMBEDDING_MODEL | The name of the embedding model to use. This determines the vector dimension. | nomic-embed-text | internal/search/service.go:14, internal/embeddings/ollama.go:28 |
OLLAMA_URL | The URL for the Ollama service that generates embeddings. | http://localhost:11434 | internal/embeddings/ollama.go:23 |
The validate() method within the client ensures that QDRANT_URL and QDRANT_COLLECTION are not empty before making any API calls.
Sources: internal/qdrant/client.go:82-91
Qdrant Client Architecture
The qdrant.Client is a struct that holds the connection details and provides methods to interact with the Qdrant REST API.
Client Structure
The diagram below shows the structure of the qdrant.Client and its methods.
This diagram illustrates the main components of the Qdrant client, including its properties and the public methods available for database interaction.
Sources: internal/qdrant/client.go:70-80, internal/qdrant/client.go:100-198
Initialization
A new client is instantiated via the qdrant.New() function. This function reads the necessary environment variables (QDRANT_URL, QDRANT_API_KEY, QDRANT_COLLECTION) and populates a Client struct.
// internal/qdrant/client.go:73-80
func New() *Client {
_ = godotenv.Load()
return &Client{
BaseUrl: os.Getenv("QDRANT_URL"),
APIKey: os.Getenv("QDRANT_API_KEY"),
Collection: os.Getenv("QDRANT_COLLECTION"),
}
}
Core Operations
The client supports three main operations: creating a collection, upserting points, and searching for points.
1. Collection Management
Before any data can be stored, a collection must exist in Qdrant. The CreateCollectionIfNotExists method handles this by sending a PUT request to the /collections/{name} endpoint.
The vector size (vecSize) is a critical parameter determined by the selected embedding model. The search.Init() function maps the model name to its corresponding vector dimension.
| Model Name | Vector Dimension |
|---|---|
nomic-embed-text | 768 |
all-minilm | 384 |
mxbai-embed-large | 512 |
The distance metric is hardcoded to Cosine similarity. The method is idempotent; if the collection already exists (HTTP 409), it returns nil without error.
Sources: internal/qdrant/client.go:100-121, internal/search/service.go:18-29
2. Data Ingestion (Upsert)
The ingestion process converts text notes into vector embeddings and stores them in Qdrant as "points". The search.IngestNotes function orchestrates this flow.
The following sequence diagram shows the steps involved in ingesting a note.
This diagram illustrates the flow from the main application entry point through the search service, which calls the embedding service and finally the Qdrant client to store the data.
Sources: cmd/app/main.go:15-17, internal/search/service.go:34-58, internal/embeddings/ollama.go:19-64
Point Data Structure
Each item stored in Qdrant is a Point, which consists of an ID, a vector, and an optional payload.
| Field | Type | Description |
|---|---|---|
ID | interface{} | A unique identifier for the point (e.g., an integer). |
Vector | []float32 | The vector embedding of the note text. |
Payload | map[string]any | A JSON object containing metadata, such as the original text and tags. |
Sources: internal/qdrant/client.go:28-32, internal/search/service.go:50-53
3. Data Retrieval (Search)
To find relevant notes, a query string is first converted into a vector embedding. The qdrant.Client.Search method then sends this vector to the Qdrant search endpoint.
The following sequence diagram outlines the search process.
This diagram shows how a user query is vectorized and used to find similar items in the Qdrant database.
Sources: cmd/app/main.go:21-23, internal/search/service.go:60-79, internal/qdrant/client.go:152-198
The search request body is defined by the searchRequest struct, which specifies the query vector, the number of results to return (limit), and whether to include the payload in the response.
// internal/qdrant/client.go:40-46
type searchRequest struct {
Vector []float32 `json:"vector"`
Limit int `json:"limit"`
Filter map[string]any `json:"filter,omitempty"`
WithPayload bool `json:"with_payload"`
WithVector bool `json:"with_vector"`
}
In the current implementation, WithPayload is always set to true and WithVector is false to retrieve the note's content without the large vector array.
Sources: internal/qdrant/client.go:156-161
Summary
The Qdrant integration in MauQNotes is facilitated by a clean, self-contained HTTP client in the internal/qdrant package. This client, combined with the search service, provides the complete pipeline for semantic search: configuring the database connection, ensuring the collection exists with the correct dimensions, ingesting text notes as vector points, and executing similarity searches. The system is designed to work with different embedding models, with the vector dimension being dynamically set based on the chosen model. This architecture forms the foundation of the note retrieval capabilities of the application.