What is DataHub?
Relevant source files
The following files were used as context for generating this wiki page:
DataHub is an open-source metadata platform designed for the modern data stack. It functions as a unified context graph that enables data discovery, governance, and observability for both human teams and AI agents. Originally developed at LinkedIn, it facilitates the connection of technical metadata, business context, and lineage into a single, accessible graph.
Sources: docs-website/static/llms.txt:3-7, metadata-ingestion/README.md:9-13
The platform supports both self-hosted (open-source) versions and a managed offering known as DataHub Cloud. It is released under the Apache 2.0 license and serves as a central hub for managing diverse data ecosystems through search, UI interfaces, and programmatic APIs.
Sources: docs-website/static/llms.txt:5-8, docs-website/README.md:23-28
Core Architecture and Data Flow
DataHub operates on an event-driven metadata model. Metadata enters the system through two primary methods: pull-based ingestion and push-based ingestion.
Metadata Ingestion Flow
Pull-based ingestion uses a specialized framework to crawl data systems like Snowflake, BigQuery, and dbt on a schedule. Push-based ingestion allows external applications to emit metadata directly using Python or Java SDKs.
The diagram above illustrates how metadata is extracted from various sources, processed into WorkUnits, and eventually stored in the DataHub Context Graph.
Sources: metadata-ingestion/README.md:9-15, metadata-ingestion/adding-source.md:37-45, docs-website/static/llms.txt:104-106
Key Components
| Component | Description |
|---|---|
| GMS | Generalized Metadata Service: The backend service for serving and persisting metadata. |
| MetadataWorkUnit | A container for metadata events (MCP/MCE) produced by ingestion sources. |
| MCP/MCL | Metadata Change Proposals and Logs: The event-based mechanism for updating the model. |
| DataHub UI | A React-based web application for searching, browsing, and managing metadata. |
Sources: metadata-ingestion/adding-source.md:40-45, datahub-web-react/README.md:9-11, docs-website/static/llms.txt:104-108
Core Capabilities and Features
DataHub provides a suite of tools to manage the lifecycle of data assets across an organization.
Discovery and Governance
The platform offers a full-text search experience across datasets, dashboards, and ML models. Governance is managed through Ownership, Tags, and a Business Glossary to standardize terminology.
Sources: docs-website/static/llms.txt:75-85, datahub-web-react/README.md:16-20
Observability and Quality
DataHub includes features for tracking data health through Assertions and Data Contracts. Assertions allow for automated anomaly detection and data quality checks (e.g., Freshness, Volume, and Schema checks).
Sources: docs-website/static/llms.txt:86-90, docs-website/src/learn/data-mesh.md:104-115
Data Mesh Implementation
DataHub supports the Data Mesh paradigm by allowing organizations to define Domains and Data Products. This decentralizes ownership to specific business units while maintaining federated governance.
This diagram represents the logical organization of data into Domains and Products under a federated governance model.
Sources: docs-website/src/learn/data-mesh.md:27-44, docs-website/src/learn/data-mesh.md:100-103
Technical Implementation Details
Metadata Modeling
DataHub uses a structured metadata model consisting of Entities and Aspects.
- Entities: The primary nodes in the graph (e.g., Dataset, User, Dashboard).
- Aspects: Specific attributes or properties associated with an entity (e.g., Ownership, Schema, Documentation).
Sources: docs-website/static/llms.txt:104-105, datahub-web-react/README.md:143-150
Ingestion Source Development
Developers can add new sources by extending the Python ingestion framework. This involves creating a ConfigModel (using Pydantic), a Reporter, and implementing the Source class.
# Example of a Source Configuration using Pydantic
from pydantic import Field
from datahub.api.configuration.common import ConfigModel
class CustomSourceConfig(ConfigModel):
client_id: str = Field(description="Source API client id.")
base_url: str = Field(description="Url to the source system.")
Sources: metadata-ingestion/adding-source.md:25-35, metadata-ingestion/adding-source.md:58-75
Development and Testing
The project utilizes a Gradle-based build system. For the metadata-ingestion module, setup.py serves as the source of truth for dependencies.
| Task | Command |
|---|---|
| Setup Dev Env | ../gradlew :metadata-ingestion:installDev |
| Run Fast Tests | ../gradlew :metadata-ingestion:testQuick |
| Linting | ../gradlew :metadata-ingestion:lint |
Sources: metadata-ingestion/CLAUDE.md:5-15, metadata-ingestion/CLAUDE.md:76-80
Conclusion
DataHub serves as a critical coordination layer for modern data stacks, bridging the gap between technical infrastructure and business context. By providing a unified graph and supporting both automated ingestion and manual enrichment, it enables organizations to scale their data operations while maintaining high standards of quality and governance.
Sources: docs-website/static/llms.txt:3-7, docs-website/src/learn/data-mesh.md:129-132