Introduction and Getting Started
Relevant source files
The following files were used as context for generating this wiki page:
Onyx (formerly Danswer) is an open-source Gen-AI and Enterprise Search platform designed to connect company documents, applications, and people through a modular architecture. It features a community edition (MIT licensed) and an enterprise edition, providing tools for indexing, searching, and chatting with internal knowledge bases. Sources: AGENTS.md:16-19, CLAUDE.md:16-19
The platform integrates a sophisticated backend utilizing Python, FastAPI, and Celery for asynchronous processing, paired with a Next.js frontend and search capabilities powered by the Vespa vector database. Sources: AGENTS.md:109-116, CLAUDE.md:109-116
Architecture Overview
Onyx employs a multi-service architecture coordinated via Docker and background workers. The system relies on several core external services to manage data persistence, search indexing, and task orchestration. Sources: CONTRIBUTING.md:79-85, AGENTS.md:21-25
Core Components
- Backend API: Built with Python 3.11 and FastAPI to handle requests and orchestrate services.
- Frontend: A Next.js 15+ application providing the user interface for search and administration.
- Model Server: Runs local NLP models for embeddings and processing.
- Vespa: Acts as the primary vector database and search engine for document retrieval.
- PostgreSQL: Stores relational data, task states, and system metadata.
- Redis: Used for caching and inter-process communication between Celery workers.
- MinIO/S3: Provides object storage for file persistence and snapshots.
Sources: AGENTS.md:109-115, CONTRIBUTING.md:155-159, CLAUDE.md:109-115
System Data Flow
The following diagram illustrates the high-level data flow from external data sources into the search index.
The diagram shows the pipeline where the Docfetching worker retrieves raw data, the Docprocessing worker handles indexing logic, and chunks are embedded via the Model Server before storage in Vespa. Sources: AGENTS.md:32-48, CLAUDE.md:32-48
Development Environment Setup
Setting up Onyx for development requires Python 3.11, Node.js v22, and Docker for managing external services. Sources: CONTRIBUTING.md:94-100
Backend Requirements
Developers are encouraged to use uv for environment management.
uv venv .venv --python 3.11
source .venv/bin/activate
uv sync --all-extras
uv run playwright install
Sources: CONTRIBUTING.md:103-112
Frontend Requirements
The frontend is managed via npm within the web/ directory.
nvm install 22 && nvm use 22
cd web && npm i
npm run dev
Sources: CONTRIBUTING.md:117-123, web/README.md:11-13
External Services
External dependencies (Postgres, Vespa, Redis, MinIO) are typically run as Docker containers during development.
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d index relational_db cache minio
Sources: CONTRIBUTING.md:147-152
Background Task Processing (Celery)
Onyx uses specialized Celery workers to handle resource-intensive or asynchronous operations. All workers utilize thread pools rather than processes to ensure system stability. Sources: AGENTS.md:21-25, CLAUDE.md:21-25
Worker Types and Responsibilities
| Worker Type | Responsibilities | Concurrency |
|---|---|---|
| Primary | Connector management, document sync, LLM updates. | 4 threads |
| Docfetching | Fetches documents from external connectors. | Env-defined |
| Docprocessing | Chunks, embeds, and writes to Vespa/Postgres. | Env-defined |
| Light | Fast operations: Vespa metadata sync, permissions. | High |
| Heavy | Resource-intensive tasks like document pruning. | 4 threads |
| Beat | Periodic scheduler (indexing/pruning checks). | N/A |
Sources: AGENTS.md:27-77, CLAUDE.md:27-77
Connector System
Connectors are the primary mechanism for ingesting data into Onyx. They follow three main patterns: Load (bulk), Poll (incremental), and Slim (ID-only for pruning). Sources: backend/onyx/connectors/README.md:13-26
Connector Configuration Parameters
Connectors are configured via the frontend using the connectorConfigs data structure.
| Connector | Key Parameters | Frequency Override |
|---|---|---|
| Web | base_url, web_connector_type (recursive/single/sitemap) | 24 Hours |
| GitHub | repo_owner, include_prs, include_issues | Default |
| Google Drive | include_shared_drives, include_my_drives | Default |
| Confluence | wiki_base, is_cloud, space, page_id | Default |
Sources: web/src/lib/connectors/connectors.tsx:103-317
Engineering Best Practices
The Onyx project maintains strict standards for code quality and system design.
- Typing: All Python and TypeScript code must be strictly typed.
mypyis used for static analysis in the backend. Sources: AGENTS.md:85, CONTRIBUTING.md:131-135 - Error Handling: Developers must raise
OnyxErrorinstead ofHTTPException. Status codes must not be hardcoded; they should be derived fromOnyxErrorCode. Sources: AGENTS.md:213-220, CLAUDE.md:213-220 - Database Operations: All database logic must be contained within
backend/onyx/dborbackend/ee/onyx/db. Direct queries outside these directories are prohibited. Sources: AGENTS.md:12, CLAUDE.md:12 - Migrations: Database changes are managed via Alembic. Standard migrations are run with
alembic upgrade head. Sources: AGENTS.md:125-130, CLAUDE.md:125-130
Integration Sequence
The following sequence shows how a developer interacts with the system to start development.
Sources: CONTRIBUTING.md:147-185
Conclusion
Getting started with Onyx involves setting up a multi-service environment where background workers, a vector database, and an API server collaborate to provide AI-driven search. By adhering to the standardized configuration and engineering practices, developers can contribute new connectors and features while maintaining system stability and performance. Sources: AGENTS.md:248-251, CONTRIBUTING.md:219-224