Data Flow

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/app/main.go
• internal/embeddings/ollama.go
• internal/embeddings/openai.go
• internal/qdrant/client.go
• internal/search/service.go
• setup-ollama.md

The data flow in MauQNotes is centered around two primary processes: data ingestion and semantic search. The system is designed to take unstructured text notes, convert them into numerical vector representations (embeddings), and store them in a Qdrant vector database. This allows for efficient semantic querying, where a user's search query is also converted into a vector to find the most contextually similar notes.

The core components facilitating this flow are the search service, which orchestrates the operations; the embeddings service, which handles the conversion of text to vectors via external models like Ollama or OpenAI; and the qdrant client, which communicates with the Qdrant vector database for storage and retrieval. The entire process is initiated from the application's main entry point, which demonstrates a simple workflow of ingesting a set of notes and then performing a query against them.

Sources: cmd/app/main.go:15-29, internal/search/service.go:40-84

System Architecture

The application's architecture is composed of a main application logic that uses a search service to interact with an embeddings generator and a qdrant vector database client.

The diagram below illustrates the high-level interaction between these components.

Configuration

The system is configured via environment variables, which are loaded from a .env file using the godotenv package. These variables define connection details for Qdrant, the embedding model to use, and API keys.

Variable	Description	Default Value	Source File
QDRANT_URL	The base URL for the Qdrant instance.	(none)	internal/qdrant/client.go:76
QDRANT_API_KEY	The API key for authenticating with Qdrant.	(none)	internal/qdrant/client.go:77
QDRANT_COLLECTION	The name of the collection in Qdrant to use.	(none)	internal/qdrant/client.go:78
EMBEDDING_MODEL	The name of the embedding model to use.	nomic-embed-text	internal/search/service.go:13, internal/embeddings/ollama.go:31
OLLAMA_URL	The URL for the Ollama API service.	http://localhost:11434	internal/embeddings/ollama.go:24
OPENAI_API_KEY	The API key for the OpenAI service.	(none)	internal/embeddings/openai.go:26

Sources: internal/qdrant/client.go:74-81, internal/search/service.go:11-15, internal/embeddings/ollama.go:22-33, internal/embeddings/openai.go:24-29

Embedding Models

The choice of EMBEDDING_MODEL determines the dimension of the vectors. The system maps model names to their corresponding vector dimensions.

Model Name	Vector Dimension	Source
nomic-embed-text	768	internal/search/service.go:22, setup-ollama.md:21
all-minilm	384	internal/search/service.go:24, setup-ollama.md:22
mxbai-embed-large	512	internal/search/service.go:26

Note: setup-ollama.md lists mxbai-embed-large as having 1024 dimensions, while search/service.go implements it as 512. The code in search/service.go takes precedence for runtime behavior.

Sources: internal/search/service.go:18-31, setup-ollama.md:21-23

Data Ingestion Flow

Data ingestion is the process of converting text notes into vector embeddings and storing them in the Qdrant database. This flow is initiated by calling search.IngestNotes.

The following sequence diagram details the steps involved in the ingestion process.

This diagram shows the IngestNotes function orchestrating the creation of a Qdrant collection, generating embeddings for each note via the Ollama service, and finally upserting the resulting vector points into the database.

Sources: cmd/app/main.go:15-17, internal/search/service.go:40-69, internal/qdrant/client.go:115-150, internal/embeddings/ollama.go:19-68

Qdrant Point Structure

The data is stored in Qdrant as Point objects. Each point consists of a unique ID, the vector embedding, and a payload containing the original text and other metadata.

// Source: internal/qdrant/client.go:29-33
type Point struct {
	ID      interface{}    `json:"id"`
	Vector  []float32      `json:"vector"`
	Payload map[string]any `json:"payload,omitempty"`
}

In the IngestNotes function, the payload is structured as follows:

{
  "text": "The original note text.",
  "tags": ["personal", "note"]
}

Sources: internal/qdrant/client.go:29-33, internal/search/service.go:58-61

Query Flow

The query flow enables semantic search. A text query is converted into a vector, which is then used to find the most similar vectors (and their corresponding notes) in the Qdrant database. This process is handled by the search.Query function.

The sequence diagram below outlines the query process.

This diagram shows the Query function first generating an embedding for the search query, then using the qdrant.Client to perform a search against the database. A simple reranking logic is applied to the results before they are returned.

Sources: cmd/app/main.go:20-22, internal/search/service.go:71-84, internal/qdrant/client.go:152-191, internal/embeddings/ollama.go:19-68

Reranking Logic

After retrieving results from Qdrant, the search.Query function applies a "naive rerank" to boost the score of results where the original note text contains the query string as a substring (case-insensitive). If a match is found, the score is increased by 0.01.

Sources: internal/search/service.go:85-93

Core Components

Qdrant Client

The qdrant.Client in internal/qdrant/client.go is a lightweight HTTP client for interacting with the Qdrant API.

This diagram shows the structure of the Client and its main methods for managing collections, upserting data points, and performing searches.

Sources: internal/qdrant/client.go:69-191

Embedding Services

The application can generate embeddings using two different services, defined in the embeddings package. The primary implementation used by the search service is FromOllama.

• FromOllama(text string): Connects to a local Ollama instance (defaulting to http://localhost:11434) to generate embeddings. It sends a POST request to the /api/embeddings endpoint.
• FromOpenAI(text string): An alternative implementation that connects to the https://api.openai.com/v1/embeddings endpoint to generate embeddings using an OpenAI model.

Both functions take a string as input and return a []float32 vector, abstracting the specific API details from the calling service.

Sources: internal/embeddings/ollama.go:19-77, internal/embeddings/openai.go:20-69, internal/search/service.go:50,73