DataHub Upgrades & System Update
Relevant source files
The following files were used as context for generating this wiki page:
DataHub employs a structured approach to upgrades and system updates across its various components, including metadata ingestion, integration modules, and documentation systems. The process ensures that dependencies remain current while maintaining custom patches and platform-specific configurations. Upgrades typically involve a combination of automated scripts, manual verification, and rigorous lock-file management to prevent environment drift.
The system encompasses processes for upgrading third-party integrations (like OpenLineage), managing Python dependency chains through setup.py and uv.lock, and synchronizing documentation across the open-source and managed (Cloud) offerings.
Sources: metadata-ingestion/CLAUDE.md, metadata-integration/java/acryl-spark-lineage/scripts/upgrade-openlineage.sh, docs-website/generateDocsDir.ts
Metadata Ingestion Dependency Management
The metadata ingestion framework uses setup.py as the primary source of truth for all dependencies, extras, and entry points. Updates to the system require a strict synchronization process to maintain consistency between the human-readable configuration and the machine-locked environment.
Dependency Synchronization Flow
When a dependency is updated or a new ingestion source entry point is added, the project follows a "regenerate and verify" pattern.
This flow ensures that any change in setup.py is propagated through the entire chain. CI specifically runs scripts/verify_pyproject_equivalence.py to prevent manual edits to generated files like pyproject.toml.
Sources: metadata-ingestion/CLAUDE.md:83-100
Key Dependency Commands
| Action | Command | Description |
|---|---|---|
| Update Locks | ../gradlew :metadata-ingestion:updateLockFile | Syncs setup.py to uv.lock and constraints.txt. |
| Verify Sync | ../gradlew :metadata-ingestion:checkLockFile | Validates that all generated lock files match setup.py. |
| Dev Setup | ../gradlew :metadata-ingestion:installDev | Sets up the local Python environment for development. |
Sources: metadata-ingestion/CLAUDE.md:86-94, metadata-ingestion/developing.md:15-18
Integration Module Upgrades (OpenLineage Example)
Upgrading core integration libraries, such as OpenLineage within the Spark lineage module, involves a complex process of fetching upstream changes and re-applying DataHub-specific customizations. This is automated via scripts that manage "vendored" code.
Automated Upgrade Logic
The upgrade-openlineage.sh script automates the transition between versions by comparing upstream files and attempting to apply existing patches.
This logic allows the system to remain compatible with upstream improvements while preserving unique DataHub features, such as specific visitors for Spark execution plans.
Sources: metadata-integration/java/acryl-spark-lineage/scripts/upgrade-openlineage.sh:11-45
Manual Conflict Resolution
When upstream changes are too significant for automated patching, the system provides a backup directory (patches/backup-<timestamp>) for manual three-way merges. Developers must then use ./scripts/generate-patches.sh to update the customization set for the new version.
Sources: metadata-integration/java/acryl-spark-lineage/scripts/upgrade-openlineage.sh:176-192
Documentation and UI Updates
Updates to the system often require synchronized documentation changes. DataHub uses a documentation assembly script (generateDocsDir.ts) that transforms markdown files across the repository into a structured format for Docusaurus.
Documentation Generation Features
| Feature | Implementation | Source File |
|---|---|---|
| Release History | Fetches data via Octokit from GitHub releases. | generateDocsDir.ts:446 |
| Inline Snippets | {{ inline /path/to/file }} directive. | generateDocsDir.ts:358 |
| Cmd Injection | {{ command-output cmd }} embeds CLI output. | generateDocsDir.ts:402 |
| AI Mirroring | Serves cleaned .md files for LLM/AI tools. | generateDocsDir.ts:35 |
Sources: docs-website/generateDocsDir.ts:35, 358, 402, 446, docs-website/sidebars.js
Managed vs. Open Source Updates
The system handles differences between DataHub Core (Open Source) and DataHub Cloud (SaaS) using a FeatureAvailability component. This allows the same documentation source to serve different deployment models by conditionally displaying "Cloud Only" icons and notes.
Sources: docs-website/generateDocsDir.ts:635-655, docs-website/sidebars.js:68-75
System Update Summary
Updates in DataHub are a multi-layered process ensuring that the metadata ingestion framework, Spark integrations, and documentation systems move forward in unison. By centralizing dependency management in setup.py and automating the application of patches for vendored integrations, the project maintains high technical standards while allowing for rapid iteration on both core and cloud-specific features.
Sources: metadata-ingestion/CLAUDE.md, metadata-integration/java/acryl-spark-lineage/scripts/upgrade-openlineage.sh, docs-website/generateDocsDir.ts