Model Server (Embeddings & Reranking)

Relevant source files

The following files were used as context for generating this wiki page:

• CONTRIBUTING.md
• AGENTS.md
• backend/onyx/chat/README.md
• backend/onyx/prompts/search_prompts.py
• backend/onyx/connectors/README.md

The Model Server is a specialized microservice within the Onyx architecture responsible for hosting and executing local Natural Language Processing (NLP) models. Its primary functions include generating vector embeddings for document chunks and performing cross-encoder reranking to optimize search relevance. By offloading these compute-intensive tasks to a dedicated server, Onyx maintains a modular architecture that supports both local and cloud-based AI/ML workflows.

The service is built using FastAPI and typically operates on port 9000. It is a critical component of both the asynchronous indexing pipeline (managed by Celery workers) and the synchronous Retrieval-Augmented Generation (RAG) pipeline used during user chat sessions.

System Architecture and Role

The Model Server acts as an internal API for the backend services. It is utilized by the Docprocessing Worker during the ingestion phase and by the API server during the query phase.

Indexing Pipeline Integration

When documents are fetched from external connectors, they are processed through an indexing pipeline. The Model Server is invoked after document chunking to transform text into numerical vectors (embeddings) before they are written to the Vespa vector database.

Search and RAG Pipeline

During a chat session, the Model Server is used twice in the retrieval process:

1. Bi-Encoder Inference: The user's query (often rephrased by an LLM) is sent to the Model Server to generate a query embedding for semantic search in Vespa.
2. Cross-Encoder Reranking: After Vespa returns the top candidate documents, the Model Server reranks them using a more computationally expensive cross-encoder model to ensure the most relevant context is provided to the LLM.

Operational Configuration

The Model Server is designed for local development and high-performance production deployments. It is typically managed as a standalone service within the Docker Compose stack.

Development Setup

For development, the Model Server can be run manually using uvicorn.

Parameter	Value	Description
Default Port	9000	The local port the server listens on.
Command	uvicorn model_server.main:app	The entry point for the FastAPI application.
Environment	PYTHONPATH=backend	Required for local module resolution.

Sources: CONTRIBUTING.md:158-166

Core Components

• Encoders: Logic residing in encoders.py that interfaces with libraries like sentence-transformers or transformers to load models into memory (GPU or CPU) and perform batch inference.
• Reranker: Handles cross-encoder logic where pairs of (Query, Document) are scored together to determine relevance more accurately than vector similarity alone.

Data Structures and Interfaces

The Model Server interacts with the backend using specific data models defined in Pydantic.

Embedding Generation

The backend sends batches of text strings to the /encoder/bi-encoder endpoint. The server returns a list of floating-point arrays representing the document vectors.

Reranking Logic

The cross-encoder endpoint accepts the original query and a list of document blurbs. It outputs a score for each document, which the backend uses to filter or reorder results before LLM ingestion. This step is crucial for maintaining high precision in the Retrieval-Augmented Generation (RAG) flow.

Sources: backend/onyx/chat/README.md:144-149, AGENTS.md:175-179

Conclusion

The Model Server is the primary engine for local AI tasks in Onyx, enabling sophisticated semantic search and document ranking without relying solely on external LLM providers for the retrieval phase. By centralizing embedding and reranking logic, the system achieves lower latency for search operations and provides flexibility in model selection for different performance and accuracy requirements.