Search Service

Relevant source files

The following files were used as context for generating this wiki page:

• internal/search/service.go
• cmd/app/main.go
• internal/qdrant/client.go
• internal/embeddings/ollama.go
• internal/embeddings/openai.go
• setup-ollama.md

The Search Service is a core module within MauQNotes responsible for enabling semantic search capabilities. Its primary functions are to ingest textual notes by converting them into vector embeddings and storing them in a Qdrant vector database, and to query this database to find the most relevant notes based on a user's search query. It orchestrates interactions between the main application, an embedding generation service (currently Ollama), and the Qdrant database client.

Sources: internal/search/service.go, cmd/app/main.go

Architecture Overview

The service acts as a middle layer, coordinating the process of vectorization and database interaction. The main application calls the search package's functions, which in turn use the embeddings package to generate vectors and the qdrant package to interact with the database.

The following diagram illustrates the high-level architecture of the Search Service and its dependencies.

Configuration

The service is configured using environment variables, which are loaded via godotenv.

Variable	Description	Default Value	Source File
EMBEDDING_MODEL	Specifies the embedding model to use.	nomic-embed-text	internal/search/service.go:12
QDRANT_URL	The base URL for the Qdrant database instance.	(none)	internal/qdrant/client.go:76
QDRANT_API_KEY	The API key for authenticating with Qdrant.	(none)	internal/qdrant/client.go:77
QDRANT_COLLECTION	The name of the collection within Qdrant to use.	(none)	internal/qdrant/client.go:78
OLLAMA_URL	The base URL for the Ollama API service.	http://localhost:11434	internal/embeddings/ollama.go:23
OPENAI_API_KEY	The API key for authenticating with OpenAI.	(none)	internal/embeddings/openai.go:24

Core Processes

Initialization

Before any operation, the service can be initialized to determine the correct vector dimension based on the selected EMBEDDING_MODEL. The Init() function reads the model name from the environment and returns the corresponding dimension size. This is crucial for creating the Qdrant collection with the correct vector schema.

Sources: internal/search/service.go:10-33

The supported models and their dimensions are:

Model Name	Vector Dimension
nomic-embed-text	768
all-minilm	384
mxbai-embed-large	512

Sources: internal/search/service.go:18-27, setup-ollama.md:21-25

Ingestion Flow

The IngestNotes function handles the process of adding new notes to the search index.

1. A new Qdrant client is created.
2. The service is initialized to get the vector dimension.
3. A Qdrant collection is created if it doesn't already exist, using the determined dimension and Cosine distance metric.
4. For each note string, an embedding is generated via embeddings.FromOllama.
5. A qdrant.Point is constructed with a unique ID, the generated vector, and a payload containing the original text.
6. All points are sent to Qdrant in a single Upsert operation.

Sources: internal/search/service.go:35-61

The sequence diagram below details the ingestion flow.

Query Flow

The Query function performs a semantic search.

1. A new Qdrant client is created.
2. An embedding is generated for the query string q using embeddings.FromOllama.
3. The Qdrant client's Search method is called with the query vector and the desired number of results (topK).
4. A naive reranking is applied: if a result's payload text contains the query string (case-insensitive), its score is boosted by 0.01.
5. The potentially reranked results are returned.

Sources: internal/search/service.go:63-85

The sequence diagram below details the query flow.

Key Components

Qdrant Client

The qdrant.Client is a struct that encapsulates the logic for communicating with the Qdrant database's HTTP API. It manages the base URL, API key, and collection name.

Sources: internal/qdrant/client.go:69-74

Key methods include:

• New(): Factory function to create a new client from environment variables.
• CreateCollectionIfNotExists(vecSize, distance): Sends a PUT request to create a collection, ignoring 409 Conflict errors if it already exists.
• Upsert(points): Sends a PUT request to add or update points in the collection.
• Search(queryVec, limit): Sends a POST request to perform a vector search.

The diagram below shows the Client and its related data structures.

Embeddings Package

The embeddings package provides functions to convert text into vector embeddings by calling external model APIs. While two providers are implemented, the Search Service currently uses only FromOllama.

• FromOllama(text string): Sends a request to the Ollama API endpoint (/api/embeddings) with the specified model and prompt. It returns the embedding as a []float32 slice.
  Sources: internal/embeddings/ollama.go:20-68
• FromOpenAI(text string): Sends a request to the OpenAI API endpoint (/v1/embeddings). This function is available but not currently used by the search service.
  Sources: internal/embeddings/openai.go:23-69