Feature: Indexing Notes

Relevant source files

The following files were used as context for generating this wiki page:

• cmd/app/main.go
• internal/search/service.go
• internal/qdrant/client.go
• internal/embeddings/ollama.go
• internal/embeddings/openai.go
• setup-ollama.md

The note indexing feature is the core component responsible for processing textual notes, converting them into numerical vector representations (embeddings), and storing them in a Qdrant vector database. This process, known as ingestion, is the foundational step for enabling semantic search capabilities within the application. By transforming text into vectors, the system can later perform similarity searches based on meaning rather than just keywords.

The main entry point for this feature is the search.IngestNotes function, which orchestrates the entire workflow from receiving raw text to upserting the final data points into the database. The system is designed to be configurable, supporting different embedding models and providers like Ollama and OpenAI.

Sources: cmd/app/main.go:15-17, internal/search/service.go:37-62

Architecture Overview

The indexing process involves several key components that work together to ingest notes. The search service acts as the central coordinator, utilizing an embeddings service to generate vectors and a qdrant client to communicate with the vector database.

The following diagram illustrates the high-level data flow for note ingestion.

This diagram shows the flow starting from the main application, through the search and embedding services, to the final interaction with the Qdrant database.

Sources: cmd/app/main.go:15, internal/search/service.go:37-62, internal/embeddings/ollama.go:19, internal/qdrant/client.go:110,133

Detailed Ingestion Process

The ingestion process is handled by the IngestNotes function in the search package. It follows a clear sequence of steps to ensure notes are correctly processed and stored.

Sources: internal/search/service.go:37-62

Ingestion Sequence

The sequence diagram below details the interactions between the different components during the IngestNotes call.

This diagram shows the step-by-step process: initialization, collection creation, looping through notes to generate embeddings, and finally, batch upserting the generated points into Qdrant.

Sources: internal/search/service.go:37-62, internal/qdrant/client.go:110-153

Data Structure for Indexing

The data is stored in Qdrant as Point objects. Each point consists of a unique ID, the generated vector, and a payload containing the original text and associated metadata.

// internal/qdrant/client.go:30-34
type Point struct {
	ID      interface{}    `json:"id"`
	Vector  []float32      `json:"vector"`
	Payload map[string]any `json:"payload,omitempty"`
}

An example payload includes the note's text and a list of tags.

// internal/search/service.go:52-55
Payload: map[string]any{
    "text": n,
    "tags": []string{"personal", "note"},
},

The relationship between a note and its stored representation in Qdrant can be visualized as follows.

Embedding Generation

The system supports generating embeddings via two providers: a local Ollama instance or the remote OpenAI API. The choice of provider and model is determined by environment variables. The primary function used in the ingestion flow is embeddings.FromOllama.

Sources: internal/search/service.go:47, internal/embeddings/ollama.go, internal/embeddings/openai.go

Ollama Provider

The FromOllama function connects to a local Ollama service to generate embeddings.

• Endpoint: It sends a POST request to {OLLAMA_URL}/api/embeddings.
• Default URL: If OLLAMA_URL is not set, it defaults to http://localhost:11434.
• Model: It uses the model specified in the EMBEDDING_MODEL environment variable, defaulting to nomic-embed-text.
• Request Body:

  {
    "model": "nomic-embed-text",
    "prompt": "your note text here"
  }
  

• Output: It parses the response to extract the embedding array and converts it from []float64 to []float32.

Sources: internal/embeddings/ollama.go:19-70

OpenAI Provider

Although not directly used in the IngestNotes flow, the FromOpenAI function provides an alternative for embedding generation.

• Endpoint: It sends a POST request to https://api.openai.com/v1/embeddings.
• Authentication: It uses a Bearer token with the OPENAI_API_KEY.
• Model: It uses the model specified in the EMBEDDING_MODEL environment variable.
• Request Body:

  {
    "model": "your-openai-model",
    "input": ["your note text here"]
  }
  

• Output: It extracts the embedding from the response data and converts it to []float32.

Sources: internal/embeddings/openai.go:23-64

Configuration

The indexing feature is configured through environment variables, typically loaded from a .env file.

Variable	Description	Default Value	Source File(s)
EMBEDDING_MODEL	The name of the embedding model to use (e.g., from Ollama or OpenAI).	nomic-embed-text	internal/search/service.go:12 internal/embeddings/ollama.go:31 internal/embeddings/openai.go:26
QDRANT_URL	The base URL for the Qdrant database instance.	(none)	internal/qdrant/client.go:72
QDRANT_API_KEY	The API key for authenticating with Qdrant Cloud.	(none)	internal/qdrant/client.go:73
QDRANT_COLLECTION	The name of the collection in Qdrant to store the notes.	(none)	internal/qdrant/client.go:74
OLLAMA_URL	The URL for the Ollama API service.	http://localhost:11434	internal/embeddings/ollama.go:23
OPENAI_API_KEY	The API key for authenticating with the OpenAI API.	(none)	internal/embeddings/openai.go:25

Sources: internal/search/service.go:11-14, internal/qdrant/client.go:71-75, internal/embeddings/ollama.go:22-32, internal/embeddings/openai.go:24-28

Supported Models and Dimensions

The vector dimension for the Qdrant collection is determined by the selected EMBEDDING_MODEL.

Model Name	Vector Dimension	Provider
nomic-embed-text	768	Ollama
all-minilm	384	Ollama
mxbai-embed-large	512	Ollama

Note: The mxbai-embed-large model is listed with 1024 dimensions in setup-ollama.md but 512 dimensions in internal/search/service.go. The code in service.go will use 512.

Sources: internal/search/service.go:19-27, setup-ollama.md:21-25

Summary

The note indexing feature is a robust system for converting text into searchable vector embeddings. It leverages external services like Ollama for embedding generation and Qdrant for efficient vector storage. The process is coordinated by the search service, which prepares data and manages the ingestion pipeline. This feature is critical for the application's semantic search functionality, transforming unstructured text notes into a structured, queryable format.

Sources: cmd/app/main.go, internal/search/service.go