Configuring Local Models (Ollama)

Relevant source files

The following files were used as context for generating this wiki page:

• providers/openai_compatible.py
• tests/test_ollama_custom_url.py
• conf/custom_models.json
• README.md
• AGENTS.md

Zen MCP Server provides robust support for integrating local Large Language Models, such as those served by Ollama. This allows developers to leverage models running on their own hardware for complete privacy, offline capability, and zero API costs. The integration is achieved through a generic "Custom Provider" designed to connect to any OpenAI-compatible API endpoint, which Ollama provides.

This document outlines the architecture and configuration steps required to connect Zen MCP Server to a local Ollama instance.

Sources: README.md:19-20, 71, slides/slide6.html:56-58

Architecture Overview

The connection to local models is facilitated by the OpenAICompatibleProvider. This provider acts as a bridge between the Zen MCP Server and any API endpoint that adheres to the OpenAI API specification. When a request is made to a model managed by this provider, it forwards the request to the configured local URL, such as http://localhost:11434 for Ollama.

The following diagram illustrates the request flow from a user's prompt to the local model and back.

Sources: providers/openai_compatible.py, tests/test_ollama_custom_url.py

Configuration

Configuration involves two main parts: setting environment variables to point to the local model server and defining the model's capabilities in a JSON file.

Environment Variables

The OpenAICompatibleProvider is activated and configured using environment variables, typically set in an .env file at the project root.

Variable	Description	Example
CUSTOM_API_URL	The base URL of the OpenAI-compatible API endpoint. For Ollama, this is usually http://localhost:11434.	http://localhost:11434
CUSTOM_MODEL_NAME	The default model identifier to use with the custom provider if no specific model is requested.	llama3.2

These variables are read by the provider upon initialization to establish the connection details.

Sources: providers/openai_compatible.py:31-32, tests/test_ollama_custom_url.py:16-17, AGENTS.md:65-66

The server startup process, including reading these variables, is shown below.

Sources: providers/openai_compatible.py:28-46, conf/custom_models.json, AGENTS.md:65-66

Model Definitions (custom_models.json)

While environment variables point to the server, the specific capabilities of the models it serves are defined in conf/custom_models.json. This file contains an array of model objects, each detailing properties like context window size, token limits, and supported features.

custom_models.json Structure

The file contains a models array. Each object in the array defines a single model.

Field	Type	Description
model_name	string	The unique identifier for the model (e.g., 'llama3.2').
aliases	array[string]	A list of short names for the model.
context_window	integer	The total number of tokens the model can process.
max_output_tokens	integer	The maximum number of tokens the model can generate.
supports_extended_thinking	boolean	Whether the model supports extended reasoning tokens.
supports_json_mode	boolean	Whether the model can guarantee valid JSON output.
supports_function_calling	boolean	Whether the model supports function/tool calling.
supports_images	boolean	Whether the model can process image inputs.
max_image_size_mb	float	Maximum total size in MB for all images.
description	string	A human-readable description of the model.
intelligence_score	integer	A 1-20 rating used for ordering in auto-mode.

Sources: conf/custom_models.json:5-22

Example Model Definition

Here is an example definition for a local llama3.2 model from the configuration file.

{
  "models": [
    {
      "model_name": "llama3.2",
      "aliases": [
        "local-llama",
        "ollama-llama"
      ],
      "context_window": 128000,
      "max_output_tokens": 64000,
      "supports_extended_thinking": false,
      "supports_json_mode": false,
      "supports_function_calling": false,
      "supports_images": false,
      "max_image_size_mb": 0.0,
      "description": "Local Llama 3.2 model via custom endpoint (Ollama/vLLM) - 128K context window (text-only)",
      "intelligence_score": 6
    }
  ]
}

Sources: conf/custom_models.json:23-41

Provider Implementation

The OpenAICompatibleProvider class in providers/openai_compatible.py is responsible for the integration.

Initialization

During its initialization, the provider performs these key actions:

1. Reads CUSTOM_API_URL and CUSTOM_MODEL_NAME from the environment.
2. If CUSTOM_API_URL is not set, the provider is disabled.
3. Loads the model definitions from conf/custom_models.json.
4. Creates ModelCapability objects for each model defined in the JSON file and registers them with the server.

# providers/openai_compatible.py

class OpenAICompatibleProvider(BaseProvider):
    def __init__(self, registry: "ProviderRegistry"):
        super().__init__(registry)
        self.provider_name = "custom"
        self.api_url = os.environ.get("CUSTOM_API_URL")
        self.default_model = os.environ.get("CUSTOM_MODEL_NAME")

        if not self.api_url:
            logger.info("CUSTOM_API_URL not set, disabling custom provider.")
            self.enabled = False
            return

        self.enabled = True
        self.models = self._load_models_from_json("custom_models.json")
        self._register_models()

This snippet shows the provider's constructor, where it checks for the required environment variable and loads model configurations.

Sources: providers/openai_compatible.py:28-46

Testing the Integration

The repository includes tests that validate the connection to a custom Ollama URL. The test test_custom_provider_with_ollama_url specifically sets the CUSTOM_API_URL to a mock Ollama endpoint and asserts that the OpenAICompatibleProvider is enabled and its API URL is correctly configured.

# tests/test_ollama_custom_url.py

@patch.dict(os.environ, {"CUSTOM_API_URL": "http://localhost:11434"})
def test_custom_provider_with_ollama_url(provider_registry):
    """
    Test that the Custom provider is enabled and configured correctly
    when CUSTOM_API_URL is set to an Ollama-like URL.
    """
    custom_provider = provider_registry.get_provider("custom")
    assert custom_provider is not None
    assert custom_provider.enabled
    assert custom_provider.api_url == "http://localhost:11434"

This test case demonstrates how the CUSTOM_API_URL environment variable is used to configure the provider for an Ollama endpoint.

Sources: tests/test_ollama_custom_url.py:16-26

Summary

To configure Zen MCP Server with a local Ollama model, a developer needs to:

1. Ensure an Ollama instance is running and serving a model at an accessible URL (e.g., http://localhost:11434).
2. Set the CUSTOM_API_URL environment variable to this URL in the .env file.
3. Optionally set CUSTOM_MODEL_NAME to specify a default local model.
4. Add or update the model's definition in conf/custom_models.json to reflect its capabilities, ensuring the model_name matches what Ollama uses.

By following these steps, developers can seamlessly integrate local models into their AI-powered workflows, benefiting from the privacy, speed, and cost-effectiveness of local inference.

Sources: README.md:61-68, providers/openai_compatible.py, conf/custom_models.json