Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
Onyx (formerly Danswer) is an open-source Gen-AI and Enterprise Search platform designed to connect to company documents, applications, and people. It features a modular architecture that supports both Community Edition (MIT licensed) and Enterprise Edition (EE) offerings. The platform integrates a complex backend powered by FastAPI and Celery with a modern Next.js frontend, utilizing Vespa for vector search and PostgreSQL for relational data storage.
Sources: AGENTS.md:16-19, CONTRIBUTING.md:151-158
System Component Overview
The Onyx architecture is divided into several core services that work together to provide search and AI capabilities.
Technology Stack
- Backend: Python 3.11, FastAPI, SQLAlchemy, Alembic, Celery.
- Frontend: Next.js 15+, React 18, TypeScript, Tailwind CSS.
- Data Stores: PostgreSQL (Relational), Redis (Caching/Task Queue), Vespa (Vector Database/Search), MinIO (File Store).
- AI/ML: LangChain, LiteLLM for multi-provider support.
Sources: AGENTS.md:144-152, CONTRIBUTING.md:118-125
High-Level Component Interaction
The following diagram illustrates how the primary services interact within the Onyx ecosystem.
The API Server acts as the central hub, coordinating between the frontend, databases, and the asynchronous task queue.
Sources: AGENTS.md:21-66, CONTRIBUTING.md:209-234
Backend Architecture
The backend is built as a set of microservices and background workers. All database operations are centralized under backend/onyx/db or backend/ee/onyx/db.
Background Processing (Celery)
Onyx relies heavily on Celery for asynchronous task processing. All workers use thread pools rather than processes for stability.
| Worker Type | Responsibilities | Concurrency |
|---|---|---|
| Primary | Connector management, document sync, pruning, LLM updates. | 4 threads |
| Docfetching | Fetches raw data from external connectors; spawns docprocessing tasks. | Configurable |
| Docprocessing | Chunks, embeds (via Model Server), and writes to Vespa/Postgres. | Configurable |
| Light | Lightweight tasks like permission sync and Vespa operations. | High |
| Heavy | Resource-intensive tasks, primarily document pruning. | 4 threads |
| Beat | Scheduler for periodic tasks (indexing checks every 15s, etc.). | N/A |
Sources: AGENTS.md:21-79, AGENTS.md:10-14
Data Indexing Pipeline
The indexing pipeline is the core of Onyx's search capability, moving data from external sources into the searchable vector store.
The Docprocessing worker manages the transition from raw fetched data to searchable vector embeddings.
Sources: AGENTS.md:33-43, backend/onyx/connectors/README.md:15-35
Connector Architecture
Connectors are modular components that interface with external data sources. They follow three main execution flows:
- Load Connector: Performs bulk indexing to reflect a specific point in time.
- Poll Connector: Incrementally updates the index by pulling changes since the last poll.
- Slim Connector: A lightweight method for checking if document IDs still exist, used for pruning.
Sources: backend/onyx/connectors/README.md:15-30
Frontend Architecture (Opal Design System)
The frontend is migrating to a standardized design system called Opal. It emphasizes consistency and strict adherence to Figma-defined styles.
Component Hierarchy
- Opal Primitives (
web/lib/opal/src/): Preferred for all new development. - Refresh Components (
web/src/refresh-components/): Production components awaiting migration to Opal. - Legacy Components (
web/src/components/): Deprecated components being phased out.
Sources: web/AGENTS.md:9-15
UI Layout Logic
Opal utilizes layout primitives like Content and ContentAction which automatically route to internal layouts (Xl, Lg, Md, Sm) based on sizePreset and variant props.
Sources: web/AGENTS.md:21-43
Onyx Craft (AI Coding Agent)
Onyx Craft is a specialized module that allows AI agents to build web applications, documents, and presentations in a sandboxed environment.
Sandbox Architecture
Craft supports two backends for code execution:
- Self-Hosted: Process-level isolation using filesystem-based sandboxes under
/tmp/onyx-sandboxes. - Cloud (Kubernetes): Pod-based isolation with sidecars for S3 file synchronization and session persistence.
| Component | Role |
|---|---|
| Sandbox Container | Runs the OpenCode agent and Next.js preview server. |
| File-Sync Sidecar | Handles S3 snapshots for session persistence. |
| OpenCode CLI | The agent interface using JSON-RPC 2.0 over stdin/stdout. |
Sources: web/src/app/craft/README.md:46-64, web/src/app/craft/README.md:88-91
Craft Session Lifecycle
Sources: web/src/app/craft/README.md:66-77
Security and Authentication
Onyx implements a document-level permission system.
- Permissions are synced from the source Access Control Lists (ACLs).
- During search, query-time filtering ensures users only see documents they have access to in the original system.
- Supported Auth types include OAuth2, SAML, and multi-provider support.
Sources: AGENTS.md:149, web/tests/e2e/chat/chat_message_rendering.spec.ts:48-55
Summary
The Onyx architecture is a robust, distributed system designed for high-performance enterprise search and AI interaction. By leveraging a multi-worker Celery backend, a specialized vector search pipeline using Vespa, and a modern frontend grounded in the Opal design system, it provides a scalable platform for internal knowledge management. The inclusion of Onyx Craft further extends this architecture into the realm of AI-driven application generation, supported by isolated sandbox environments.