External Permissions Syncing
Relevant source files
The following files were used as context for generating this wiki page:
External Permissions Syncing is a critical system within Onyx that ensures document access rights from source systems (e.g., Google Drive, Confluence, Slack, GitHub) are accurately reflected in the search index. This mechanism prevents unauthorized access by ensuring that when a user searches, they only see documents they have explicit permission to view in the original source application.
The system utilizes specialized background workers to periodically fetch Access Control Lists (ACLs) and group memberships. It operates through two primary avenues: Document-Level Permission Syncing (mapping users to documents) and External Group Syncing (mapping users to source-specific groups).
Sources: AGENTS.md:65-69, backend/ee/onyx/external_permissions/sync_params.py:102-171
Architecture and Worker Types
Permissions syncing is orchestrated by dedicated Celery workers that handle different granularities of data.
- Light Worker: This worker is responsible for high-concurrency, fast operations. It specifically handles document permission syncs and external group syncs.
- Docprocessing Worker: During the initial indexing of a document, this worker writes document metadata and chunks to the database and Vespa, including the initial permission set retrieved by the connector.
Sources: AGENTS.md:54-58, CLAUDE.md:54-58
Syncing Workflow
The following diagram illustrates how the system coordinates between external sources and the internal Onyx index.
The diagram shows the flow of permission data from external sources into the internal databases, ultimately filtering search results at query time.
Sources: AGENTS.md:54-58, CLAUDE.md:54-58
Core Syncing Components
The logic for permissions is defined via configuration objects that map a DocumentSource to its respective sync functions and frequencies.
Sync Configuration Models
| Model | Description | Key Fields |
|---|---|---|
DocSyncConfig | Configures how document permissions are updated. | doc_sync_frequency, doc_sync_func, initial_index_should_sync |
GroupSyncConfig | Configures how external groups are synced. | group_sync_frequency, group_sync_func, group_sync_is_cc_pair_agnostic |
SyncConfig | Container for both document and group sync settings. | doc_sync_config, group_sync_config |
Sources: backend/ee/onyx/external_permissions/sync_params.py:44-63
Connector Interfaces
Connectors implement specific interfaces to support permission syncing:
- SlimConnectorWithPermSync: A specialized connector interface that fetches document IDs and minimal permission data (SlimDocuments) rather than full document content.
- CheckpointedConnector: Allows the sync process to resume from a specific point using checkpoints, essential for large-scale data sources like Confluence.
Sources: backend/onyx/connectors/confluence/connector.py:72-80, backend/onyx/connectors/README.md:18-22
Document and Group Sync Logic
Different sources require different permission handling strategies based on their native ACL models.
Source Mapping Table
| Document Source | Doc Sync Required | Group Sync Required | CC-Pair Agnostic |
|---|---|---|---|
| Google Drive | Yes | Yes | No |
| Confluence | Yes | Yes | Yes |
| GitHub | Yes | Yes | No |
| Slack | Yes | No | N/A |
| Teams | Yes | No | N/A |
Sources: backend/ee/onyx/external_permissions/sync_params.py:102-171
Confluence Permission Syncing
Confluence permissions are fetched at the space and page level. The connector utilizes get_all_space_permissions and get_page_restrictions to build the ExternalAccess object.
def retrieve_all_slim_docs_perm_sync(
self,
start: SecondsSinceUnixEpoch | None = None,
end: SecondsSinceUnixEpoch | None = None,
callback: IndexingHeartbeatInterface | None = None,
) -> GenerateSlimDocumentOutput:
"""
Return 'slim' docs (IDs + minimal permission data).
Used primarily for incremental permission sync.
"""
return self._retrieve_all_slim_docs(
start=start,
end=end,
callback=callback,
include_permissions=True,
)
Sources: backend/onyx/connectors/confluence/connector.py:725-738
Sequence of Permission Sync
This sequence shows the Light Worker executing a scheduled task to refresh permissions for Confluence documents.
Sources: AGENTS.md:54-58, backend/onyx/connectors/confluence/connector.py:739-813
Technical Implementation Details
Permissions During Indexing
For some sources, permissions must be fetched during the initial indexing run. This is governed by initial_index_should_sync. If True, the DocSyncFuncType is invoked immediately upon the first index of the connector.
Sources: backend/ee/onyx/external_permissions/sync_params.py:47, 233-241
Slim Document Fetching
The system differentiates between a full document fetch and a "Slim" fetch for permissions. A Slim fetch only retrieves:
- Document ID: The unique identifier in the source.
- ExternalAccess: The ACL data mapping users/groups to the ID.
- Parent Hierarchy ID: Used for inheriting permissions.
Sources: backend/onyx/connectors/confluence/connector.py:772-813, web/src/lib/connectors/connectors.tsx:1072-1084
External Permissions Syncing is a foundational pillar of the enterprise search experience in Onyx. By decoupling permission fetching from content fetching through "Slim" connectors and utilizing high-concurrency background workers, the system maintains a secure and performant search index that respects complex source-system security models.