Contributing to Onyx
Relevant source files
The following files were used as context for generating this wiki page:
Onyx is an open-source Gen-AI and Enterprise Search platform designed to connect company knowledge across various data sources. Contributing to Onyx involves a structured workflow ranging from feature approval and design to implementation across a modular architecture comprising a Python backend, Next.js frontend, and specialized background workers. Sources: README.md:14-18, AGENTS.md:25-28
The project maintains a distinction between its Community Edition (MIT licensed) and Enterprise Edition (proprietary). Contributors to Enterprise Edition components (located in ee/ directories) are required to sign an IP Assignment Agreement to ensure the Company can effectively manage proprietary features. Sources: CONTRIBUTING.md:299-301, contributor_ip_assignment/EE_Contributor_IP_Assignment_Agreement.md:12-16
Contribution Process
Onyx follows a "fork and pull request" workflow. The process is divided into approval, design, and implementation phases to ensure high-quality contributions and avoid feature bloat. Sources: CONTRIBUTING.md:19-21
Workflow Phases
- Feature Approval: Contributors must create a GitHub issue to discuss the proposed enhancement. Significant features require tagging maintainers for review. Sources: CONTRIBUTING.md:23-28
- Design Review: A design document or Product Requirements Document (PRD) is required. The Onyx team may provide this or request one from the contributor depending on the feature's scope. Sources: CONTRIBUTING.md:32-34
- IP Attribution: If contributing to the Enterprise Edition (code under
ee/directories), contributors must sign the IP Assignment Agreement. Sources: CONTRIBUTING.md:36-37, contributor_ip_assignment/EE_Contributor_IP_Assignment_Agreement.md:19-21 - Review and Testing: All features must pass existing tests and address reviewer comments before merging. Sources: CONTRIBUTING.md:39-40
The diagram shows the standard lifecycle from initial proposal to the final merge into the main branch. Sources: CONTRIBUTING.md:23-40
Development Environment Setup
Onyx requires a specific stack of external software including Postgres (Relational DB), OpenSearch/Vespa (Vector DB), Redis (Cache), and MinIO (File Store). Sources: CONTRIBUTING.md:58-64
Prerequisites
- Python 3.11: Required for the backend.
- Node.js v22: Required for the frontend.
- Docker: Used to run external services.
Sources: CONTRIBUTING.md:73-77
Backend Setup
The backend uses uv for dependency management. Contributors should create a virtual environment and install dependencies with extras.
uv venv .venv --python 3.11
source .venv/bin/activate
uv sync --all-extras
uv run playwright install
Sources: CONTRIBUTING.md:81-99
Frontend Setup
The frontend is a Next.js project located in the web/ directory.
nvm use 22
cd web
npm i
npm run dev
Sources: CONTRIBUTING.md:103-109
Implementing New Connectors
Connectors are a core part of Onyx, enabling the ingestion of data from external sources. They follow specific implementation flows:
| Connector Type | Purpose | Logic |
|---|---|---|
| Load Connector | Bulk Indexing | Indexes documents to reflect a point in time (e.g., via API or dump file). |
| Poll Connector | Incremental Updates | Updates documents based on a time range to keep the index fresh. |
| Slim Connector | Pruning | Fetches only document IDs to check existence for index cleanup. |
| Event Connector | Real-time | Listens to events/webhooks for immediate updates. |
Sources: backend/onyx/connectors/README.md:12-25
Connector Architecture
A new connector must subclass specific interfaces (e.g., LoadConnector, PollConnector) and implement methods for credential loading and data fetching. Sources: backend/onyx/connectors/README.md:33-40
Connectors are instantiated via a factory that maps DocumentSource types to specific classes. Sources: backend/onyx/connectors/README.md:65-70
Background Worker System (Celery)
Onyx utilizes Celery for asynchronous processing, employing multiple specialized workers to handle specific tasks. Sources: AGENTS.md:31-33
Specialized Workers
- Primary Worker: Coordinates core background tasks like connector management and document pruning.
- Docfetching Worker: Fetches documents from external sources and monitors for stuck connectors.
- Docprocessing Worker: Handles the indexing pipeline, including chunking, embedding via the model server, and writing to the Vespa vector database.
- Light/Heavy Workers: Handle fast operations (permissions sync) and resource-intensive tasks (pruning) respectively.
- Beat Worker: The scheduler that triggers periodic tasks like indexing checks every 15 seconds.
Sources: AGENTS.md:35-85
Engineering Best Practices
Contributors are expected to adhere to specific coding standards to maintain repository health.
Backend Standards
- Typing: Everything should be as strictly typed as possible using Pydantic or domain models. Sources: CONTRIBUTING.md:183-185
- Error Handling: Always raise
OnyxErrorfromonyx.error_handling.exceptionsrather than standardHTTPException. Sources: AGENTS.md:213-217 - Async Usage: Avoid introducing new async Python code unless there is a concrete performance reason and total understanding of the event loop. Sources: CONTRIBUTING.md:225-228
Frontend (Opal Design System)
- Components: Use the Opal design system (
web/lib/opal/src/) for all new UI. Avoid legacy components inweb/src/components/. Sources: web/AGENTS.md:11-16 - Icons: Only use icons from
web/src/icons. External libraries likereact-iconsare forbidden. Sources: web/AGENTS.md:195-200 - Dark Mode: Do not use the
dark:tailwind modifier. Colors are managed viacolors.cssto ensure consistent theme inversion. Sources: web/AGENTS.md:173-178
Testing Strategy
Onyx employs a four-tier testing hierarchy to ensure system stability. Sources: AGENTS.md:139-140
| Test Level | Description | Execution Command |
|---|---|---|
| Unit | Isolated module tests with mocked external world. | pytest backend/tests/unit |
| Ext. Dependency Unit | Tests with real Postgres/Redis/Vespa but mocked internal components. | pytest backend/tests/external_dependency_unit |
| Integration | End-to-end tests against a real Onyx deployment. | pytest backend/tests/integration |
| Playwright (E2E) | Frontend and backend coordination tests. | npx playwright test |
Sources: AGENTS.md:143-188
Summary
Contributing to Onyx requires a balance between rapid iteration and architectural consistency. By following the structured approval process, adhering to the Opal design system for the frontend, and utilizing the specialized Celery worker architecture for background tasks, contributors help maintain Onyx as a high-quality, enterprise-grade AI search platform. Sources: CONTRIBUTING.md:155-160, AGENTS.md:25-28