Web Search & Connectors
Relevant source files
The following files were used as context for generating this wiki page:
Onyx utilizes a modular, connector-based architecture to bridge the gap between Large Language Models (LLMs) and distributed enterprise data. The system enables the retrieval of information from both internal company applications and the live web, feeding this data into a Retrieval-Augmented Generation (RAG) pipeline to provide grounded, citeable answers.
The core of this system is a set of specialized connectors that handle document indexing, permission syncing, and real-time updates from various sources like Google Drive, Slack, Confluence, and the open web. These connectors run as asynchronous background jobs, ensuring that the search index remains up-to-date without degrading the performance of the main application. Sources: web/tests/e2e/chat/chat_message_rendering.spec.ts:50-100, AGENTS.md:20-50
Connector Architecture
Onyx connectors are categorized by their operational flow, allowing the system to handle different data synchronization requirements effectively.
Connector Types and Flows
Connectors are implemented by subclassing specific interfaces that define how data is fetched and updated.
| Flow Type | Description |
|---|---|
| Load Connector | Performs bulk indexing to reflect a specific point in time, usually via API or dump files. |
| Poll Connector | Incrementally updates documents based on a time range to keep the index fresh. |
| Slim Connector | A lightweight method that only fetches document IDs to verify existence for pruning. |
| Event Based | Listens for real-time events to update documents (currently for future design). |
Sources: backend/onyx/connectors/README.md:10-30
Document Indexing Pipeline
When a connector fetches data, it passes through a multi-stage indexing pipeline managed by Celery workers.
The Docfetching Worker retrieves documents, while the Docprocessing Worker handles database upserts, chunking, embedding generation via the model server, and final writing to the Vespa vector database. Sources: AGENTS.md:38-55
Web Search Integration
Web search is treated as a specialized tool within the AI chat system, allowing the agent to access information beyond its training data or internal index.
Search Tool Logic
The system expansion uses tools defined in the LLM instructions to decide between internal_search and web_search. Ambiguous queries or short, keyword-heavy phrases default to internal search to prioritize company knowledge. Sources: backend/scripts/debugging/litellm/payload.json:65-90
RAG Pipeline with Web Results
When a web search is triggered, the results are streamed into the chat interface with citations.
Web results include metadata such as document_id, semantic_identifier, link, and a blurb. These are rendered in the UI as citations (e.g., [[D1]]) that link directly to the source. Sources: web/tests/e2e/chat/chat_message_rendering.spec.ts:133-170, web/tests/e2e/chat/chat_message_rendering.spec.ts:340-360
Connector Implementation & Configuration
Adding a new connector requires changes across the backend (defining the logic and factory mapping) and the frontend (defining the configuration UI).
Frontend Configuration Schema
Connector configurations are defined using a structured schema in TypeScript, supporting various input types like text, list, checkbox, and tab.
// Example configuration for the Web Connector
web: {
description: "Configure Web connector",
values: [
{
type: "text",
query: "Enter the website URL to scrape:",
label: "Base URL",
name: "base_url",
optional: false,
},
{
type: "select",
name: "web_connector_type",
options: [
{ name: "recursive", value: "recursive" },
{ name: "single", value: "single" },
{ name: "sitemap", value: "sitemap" },
],
}
]
}
Sources: web/src/lib/connectors/connectors.tsx:135-160
Security and Permissions
Onyx implements a document-level permission system. When documents are indexed from sources like Google Drive or Confluence, their original Access Control Lists (ACLs) are preserved. This ensures that during query-time filtering, users only see results they are authorized to access in the source system. Sources: web/tests/e2e/chat/chat_message_rendering.spec.ts:77-85
Background Worker Coordination
Asynchronous tasks for connectors and search are managed via Redis and Celery.
- Primary Worker: Coordinates connector management and document sync.
- Light Worker: Handles fast operations like document permission syncing.
- Heavy Worker: Handles resource-intensive tasks like document pruning.
- Beat Worker: Schedules periodic indexing checks and connector deletions.
Sources: AGENTS.md:25-80
Summary
The Web Search and Connector system in Onyx provides a unified interface for ingesting and querying diverse data. By leveraging a multi-stage indexing pipeline and a flexible tool-calling architecture for web search, the system ensures that AI responses are grounded in both internal organizational knowledge and up-to-date external information while strictly adhering to original document permissions. Sources: CONTRIBUTING.md:280-310, AGENTS.md:120-130