Home

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/app/main.go
• internal/embeddings/ollama.go
• internal/embeddings/openai.go
• internal/qdrant/client.go
• internal/search/service.go
• setup-ollama.md

MauQNotes is a semantic search application written in Go, designed to function as a personal search engine for notes. It leverages vector embeddings to understand the semantic meaning of text, allowing users to query their notes based on concepts rather than just keywords. The system integrates with embedding models via Ollama or OpenAI and uses Qdrant as its vector database for efficient storage and retrieval of high-dimensional vector data.

The core functionality is demonstrated through a simple command-line application that first ingests a set of notes by converting them into vector embeddings and storing them in Qdrant. It then performs a semantic search on these notes using a natural language query, returning the most relevant results based on vector similarity.

Sources: cmd/app/main.go, internal/search/service.go

System Architecture

The application's architecture is composed of three main layers: the application layer, the search service, and the external service clients (Embeddings and Vector DB). The main function orchestrates the two primary operations: ingestion and querying.

The overall data flow for ingestion and querying is as follows:

This diagram illustrates the two main workflows: ingesting notes and querying them, showing the interaction between the internal packages and external services.

Sources: cmd/app/main.go, internal/search/service.go, internal/embeddings/ollama.go, internal/qdrant/client.go

Core Components

Search Service

The internal/search/service.go package is the central coordinator for the application's logic. It connects the embedding generation process with the vector database operations.

Key Functions

• Init(): Initializes the service by reading the EMBEDDING_MODEL from environment variables and determining the correct vector dimension size for the Qdrant collection. It defaults to nomic-embed-text with 768 dimensions if not specified.
• IngestNotes(notes []string): Takes a slice of strings (notes), generates a vector embedding for each using the configured embedding model, creates Qdrant Point objects, and upserts them into the database.
• Query(topK int, q string): Takes a search query string, generates its vector embedding, and uses it to perform a search against the Qdrant database to retrieve the topK most similar notes. It also includes a "naive rerank" logic that slightly boosts the score of results containing the query keyword.

Sources: internal/search/service.go:12-81

Embedding Generation

Embeddings are generated by making HTTP requests to an external model provider. The application supports both Ollama for local models and the OpenAI API.

Ollama Integration

The FromOllama function in internal/embeddings/ollama.go handles the generation of embeddings using a local Ollama instance.

• Endpoint: It sends a POST request to /api/embeddings on the Ollama server.
• Configuration: The server URL is configured via OLLAMA_URL (defaults to http://localhost:11434), and the model is set by EMBEDDING_MODEL (defaults to nomic-embed-text).
• Request/Response: It sends a JSON payload with the model and prompt, and expects a JSON response containing the embedding vector.

The following sequence diagram shows the Ollama embedding generation flow:

OpenAI Integration

The FromOpenAI function in internal/embeddings/openai.go provides an alternative for generating embeddings using the OpenAI API.

• Endpoint: It sends a POST request to https://api.openai.com/v1/embeddings.
• Configuration: Requires OPENAI_API_KEY and EMBEDDING_MODEL to be set in the environment.
• Authentication: Uses a Bearer token with the OPENAI_API_KEY.

Sources: internal/embeddings/openai.go:23-69

Supported Models

The embedding model determines the dimensionality of the vectors. The application is pre-configured to handle several models.

Model Name	Vector Dimension	Provider	Notes
nomic-embed-text	768	Ollama	Recommended default, 274MB
all-minilm	384	Ollama	Smaller and faster, 137MB
mxbai-embed-large	512	Ollama	Larger and more accurate, 669MB

Note: The setup-ollama.md file lists mxbai-embed-large as having 1024 dimensions, but internal/search/service.go implements it with 512 dimensions. The code's implementation is what the application will use.

Sources: internal/search/service.go:21-33, setup-ollama.md:15-21

Vector Database Client

Interaction with the Qdrant vector database is handled by the client defined in internal/qdrant/client.go. This client is a wrapper around Qdrant's HTTP REST API.

Client and Data Structures

The Client struct holds the configuration needed to connect to Qdrant. The primary data structures map directly to the Qdrant API request/response bodies.

This diagram shows the main Client class and its relationship with the Point data structure used for upserting data.

Sources: internal/qdrant/client.go:16-177

Key Operations

1. CreateCollectionIfNotExists(vecSize int, distance Distance): Sends a PUT request to /collections/{name}. It creates a new collection with the specified vector size and distance metric (Cosine). It gracefully handles a 409 Conflict error if the collection already exists.
   Sources: internal/qdrant/client.go:101-118
2. Upsert(points []Point): Sends a PUT request to /collections/{name}/points?wait=true. This operation adds or updates points in the collection. Each point consists of an ID, a vector, and a JSON payload. The wait=true query parameter ensures the operation is completed before the API returns a response.
   Sources: internal/qdrant/client.go:120-141
3. Search(queryVec []float32, limit int): Sends a POST request to /collections/{name}/points/search. It searches for the limit number of vectors most similar to the queryVec. The request specifies with_payload: true to retrieve the original note content.
   Sources: internal/qdrant/client.go:143-177

Application Entrypoint

The main function in cmd/app/main.go serves as a working example of the system's capabilities. It defines a small list of notes, calls search.IngestNotes to process and store them, and then executes a sample query with search.Query, printing the results to the console.

The overall process flow is demonstrated in this sequence diagram:

Configuration

The application is configured using environment variables, which are loaded from a .env file by the godotenv package.

Variable	Description	Default Value	Source File(s)
EMBEDDING_MODEL	The name of the embedding model to use.	nomic-embed-text	internal/search/service.go:14, internal/embeddings/ollama.go:30
OLLAMA_URL	The base URL for the Ollama API server.	http://localhost:11434	internal/embeddings/ollama.go:24
OPENAI_API_KEY	The API key for the OpenAI service.	(none)	internal/embeddings/openai.go:24
QDRANT_URL	The base URL for the Qdrant database instance.	(none)	internal/qdrant/client.go:68
QDRANT_API_KEY	The API key for the Qdrant database (if required).	(none)	internal/qdrant/client.go:69
QDRANT_COLLECTION	The name of the collection to use in Qdrant.	(none)	internal/qdrant/client.go:70

Sources: internal/search/service.go, internal/embeddings/ollama.go, internal/embeddings/openai.go, internal/qdrant/client.go

Summary

MauQNotes provides a complete, albeit simple, implementation of a semantic search engine. It effectively demonstrates how to combine a Go application with external services like Ollama and Qdrant to build a powerful RAG (Retrieval-Augmented Generation) foundation. The separation of concerns between the search service, embedding clients, and the vector DB client makes the architecture modular and extensible. The main application acts as a clear, executable example of both the ingestion and query pipelines.