Data Schema in Qdrant
Relevant source files
The following files were used as context for generating this wiki page:The MauQNotes application utilizes Qdrant as its vector database to store and search through personal notes. The data schema within Qdrant is not rigidly defined by traditional table structures but is instead shaped by the configuration of the Qdrant "collection" and the structure of the data "points" inserted into it. This document outlines the schema, including collection parameters, the structure of data points, the payload format, and the data flow for ingestion and querying.
The core components defining the schema are the vector embeddings generated from notes and the associated metadata (payload) stored alongside them. The entire process is orchestrated by the search service, which interacts with an embedding model (like Ollama) and the Qdrant client.
Sources: internal/search/service.go, internal/qdrant/client.go
Qdrant Collection Schema
A Qdrant collection is the primary container for the note vectors. Before any data can be ingested, a collection must be created with specific vector parameters. The application ensures a collection exists by calling CreateCollectionIfNotExists.
Sources: internal/qdrant/client.go:106-125, internal/search/service.go:39-42
Collection Configuration
The collection is configured with a specific vector size (dimensionality) and a distance metric for comparing vectors.
This diagram shows the flow for ensuring a collection is created with the correct parameters before ingestion.
Sources: internal/search/service.go:37-42, internal/qdrant/client.go:106-108
The configuration is defined in the createCollectionRequest struct sent to the Qdrant API.
| Parameter | Type | Description | Source |
|---|---|---|---|
vectors.size | int | The dimensionality of the vectors. This is determined by the embedding model used. | internal/qdrant/client.go:19 |
vectors.distance | Distance (string) | The metric used to calculate similarity between vectors. It is set to Cosine. | internal/qdrant/client.go:14, internal/qdrant/client.go:20 |
The vector dimension (size) is dynamically determined based on the EMBEDDING_MODEL environment variable.
Model (EMBEDDING_MODEL) | Dimension (size) |
|---|---|
nomic-embed-text (default) | 768 |
all-minilm | 384 |
mxbai-embed-large | 512 |
Sources: internal/search/service.go:19-29, setup-ollama.md:21-23
Point Data Structure
Each item stored in the Qdrant collection is a "Point". A point consists of a unique identifier, a vector embedding, and a payload containing the original data and metadata.
This ER diagram illustrates that a Qdrant collection contains multiple points.
Sources: internal/qdrant/client.go:24-28
The Point struct defines the structure for each document inserted into Qdrant.
| Field | Go Type | JSON Key | Description |
|---|---|---|---|
ID | interface{} | id | A unique identifier for the point. In this application, it's an integer representing the note's index. |
Vector | []float32 | vector | The numerical representation (embedding) of the note text, generated by an embedding model. |
Payload | map[string]any | payload | A JSON object containing arbitrary metadata associated with the vector. |
Sources: internal/qdrant/client.go:24-28, internal/search/service.go:50-57
Payload Schema
The Payload is a flexible JSON object that stores the data associated with a vector. In MauQNotes, it is used to store the original note text and any relevant tags.
| Field | Type | Description | Example |
|---|---|---|---|
text | string | The original content of the note. | "Belajar vector database..." |
tags | []string | A list of tags associated with the note. | ["personal", "note"] |
An example of a fully constructed Point payload:
{
"text": "Go dan Qdrant cocok untuk aplikasi semantic search yang cepat.",
"tags": ["personal", "note"]
}
Sources: internal/search/service.go:54-56
Data Ingestion Flow
The process of ingesting notes involves generating embeddings and upserting them into Qdrant as points. This is handled by the IngestNotes function.
This sequence diagram shows how a note string is converted into a Qdrant point and stored.
Sources: cmd/app/main.go:15-17, internal/search/service.go:44-61, internal/qdrant/client.go:127-148, internal/embeddings/ollama.go:23-71
Search and Retrieval Schema
When a query is performed, the application generates an embedding for the query text and uses it to find the most similar points in the Qdrant collection.
Search Request
A search is initiated by sending a searchRequest to the Qdrant API.
| Field | Go Type | JSON Key | Description |
|---|---|---|---|
Vector | []float32 | vector | The embedding of the query string. |
Limit | int | limit | The maximum number of results to return (topK). |
WithPayload | bool | with_payload | Set to true to include the payload in the results. |
WithVector | bool | with_vector | Set to false as the vector itself is not needed in the result. |
Sources: internal/qdrant/client.go:31-37, internal/qdrant/client.go:154-160
Search Result
The Qdrant API returns a searchResult object, which is then processed into a more generic []map[string]any for application use.
The structure of each item in the returned result list is as follows:
| Key | Type | Description |
|---|---|---|
id | any | The unique ID of the matching point. |
score | float32 | The similarity score between the query vector and the result vector. Higher is better. |
payload | map[string]any | The payload of the matching point, containing the text and tags. |
Sources: internal/qdrant/client.go:39-46, internal/qdrant/client.go:182-189
The search flow is illustrated below:
This diagram outlines the steps from receiving a user query to returning a ranked list of matching notes.
Sources: cmd/app/main.go:20-22, internal/search/service.go:63-83, internal/qdrant/client.go:150-190
Conclusion
The data schema for MauQNotes in Qdrant is implicitly defined through a combination of collection settings and the structure of the Point object. A collection is configured with a vector dimension matching the chosen embedding model and uses Cosine similarity. Each Point encapsulates a note by pairing its vector embedding with a payload containing the original text and tags. This schema is fundamental to the application's ability to perform fast and effective semantic search on personal notes.