Ollama Local Inference
Relevant source files
The following files were used as context for generating this wiki page:
Ollama Local Inference provides the foundational AI capabilities for the Nyansa project, handling both Large Language Model (LLM) chat responses and text embeddings. It runs entirely within the self-hosted environment to ensure data privacy and offline functionality. The system manages the lifecycle of models, including automatic pulling of required models like qwen2.5:3b for chat and nomic-embed-text for vector embeddings.
Sources: README.md:16-19, README.md:214-222
System Architecture
The Ollama service sits within the internal nyansa Docker network. It interacts with three primary consumers: Open WebUI for user chat, n8n for workflow automation, and the memory-api for generating embeddings during Obsidian vault ingestion.
Network and Connectivity
Ollama publishes no ports to the host and is only reachable via the internal network at http://nyansa-ollama:11434. This isolation ensures that external entities cannot interact with the inference engine directly.
Sources: README.md:73-81, n8n/demo-data/credentials/xHuYe0MDGOs9IpBW.json:6
Component Interaction
The following diagram illustrates how different services interact with the Ollama inference engine.
The nyansa-ollama-pull-model container performs a one-shot task to ensure models are present before other services begin inference tasks.
Sources: README.md:68-81
Model Configuration
Nyansa uses specific models optimized for local execution on machines with limited RAM.
| Model | Role | Description |
|---|---|---|
qwen2.5:3b | Chat / Inference | Primary LLM for generating text and answering questions. |
nomic-embed-text | Embeddings | Generates vector representations of note chunks for RAG. |
Sources: README.md:17-19, README.md:126-127
Inference Profiles
The project provides different Docker Compose profiles to optimize Ollama performance based on available hardware:
cpu: Standard execution for machines without dedicated GPUs.gpu-nvidia: Leverages NVIDIA hardware via CUDA.gpu-amd: Leverages AMD hardware via ROCm.
Sources: README.md:19-20, README.md:120
Memory API Integration
The memory-api service uses the OllamaEmbedder class to convert Markdown note sections into vectors. This process is essential for the Retrieval-Augmented Generation (RAG) capabilities of the system.
Embedding Process
The OllamaEmbedder batches documents to optimize throughput and applies specific task prefixes required by the nomic-embed-text model.
Client Implementation Details
The OllamaEmbedder handles communication with the Ollama API using the following parameters:
- Base URL: Defaults to
http://nyansa-ollama:11434. - Batch Size: Defaults to 32 texts per call.
- Timeout: Set to 120 seconds for long embedding tasks.
- Truncation: Enabled to prevent failures when input exceeds the model's context window.
Sources: services/memory-api/src/clients/ollama.ts:8-37, services/memory-api/test/clients.test.ts:38-42
Health Monitoring and Reliability
The system includes mechanisms to monitor Ollama's availability. The /health endpoint of the memory-api checks the status of Ollama by calling its /api/version endpoint.
- Consecutive Failures: If three notes in a row fail during ingestion (e.g., due to the Ollama service being down), the ingestion process aborts to prevent mass errors.
- Error Mapping: Failures from Ollama (like 404 for missing models) are mapped to
UpstreamErrortypes and surfaced as HTTP 502 errors to the user.
Sources: services/memory-api/src/app.ts:98-106, services/memory-api/README.md:43-50, services/memory-api/test/clients.test.ts:51-60
Summary
Ollama Local Inference serves as the core intelligence engine for Nyansa. By utilizing lightweight models and specific task prefixes for embeddings, it enables efficient RAG workflows over local Obsidian vaults. The architecture prioritizes privacy and reliability through network isolation and automated model management.