Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
DataHub is an open-source metadata platform designed to power discovery, governance, and observability across modern data and AI stacks. It operates as a unified context graph that connects technical metadata, business context, lineage, and documentation into a single accessible repository for both human users and AI agents. The platform facilitates decentralized metadata management through concepts like Data Mesh while providing centralized governance standards.
Sources: docs-website/static/llms.txt:3-7, docs-website/README.md:95-103
System Architecture
DataHub follows a modular architecture consisting of several key components: the Metadata Ingestion Framework, the Metadata Serving Tier (GMS), and the React-based Frontend UI. Metadata flows from various data systems through ingestion sources into a standardized format known as the Metadata Change Event (MCE) or Metadata Change Proposal (MCP).
High-Level Component Flow
The diagram below illustrates the flow of metadata from external systems through the ingestion framework into the DataHub ecosystem and finally to the end-user interface.
Metadata Ingestion Framework
The ingestion framework is heavily inspired by Apache Gobblin, utilizing a standardized format for metadata events. It is designed to pull metadata from a variety of data systems and move it into DataHub using sources and sinks.
Framework Components
| Component | Responsibility |
|---|---|
| Sources | Pull metadata from external systems (e.g., Snowflake, BigQuery, Kafka). |
| Sinks | Consume metadata events and move them into DataHub (typically via REST). |
| Transformers | Apply logic to modify or enrich metadata during the ingestion process. |
| WorkUnits | A stream of metadata events (MCP objects) produced by the source. |
Sources: metadata-ingestion/developing.md:129-138, metadata-ingestion/adding-source.md:37-40
Ingestion Logic
The core implementation of a source revolves around the get_workunits_internal method, which yields MetadataWorkUnit objects containing MetadataChangeEventClass (MCE) or MetadataChangeProposal (MCP) events.
# Example of MCE generation from a source
def get_workunits_internal(self) -> Iterable[MetadataWorkUnit]:
# Logic to fetch metadata from source system
# Wrap MCE/MCP in a WorkUnit
yield MetadataWorkUnit(id=f"{urn}", mcp=mcp_event)
Sources: metadata-ingestion/adding-source.md:37-46
Frontend Architecture
The DataHub UI is a React-based application that serves as the primary interface for data discovery and governance. It is designed with two core guideposts: Configurability (tailoring logos, themes, and styles) and Extensibility (simplifying the addition of new entities).
Entity Registry
The EntityRegistry acts as an abstraction layer over the rendering details of specific entities (Datasets, Users, Charts, etc.). It provides a consistent way to render views associated with any entity type throughout the application.
React App Modules
- app/entity: Contains entity definitions and visual components (profiles and previews).
- app/search: Components for the full-text search experience.
- graphql: Defines queries against the
datahub-frontendserver; uses code-generation for TypeScript objects. - conf: Stores global configuration flags and placeholders.
Sources: datahub-web-react/README.md:144-177
Documentation and Build Pipeline
DataHub uses a sophisticated build pipeline to generate its documentation site, integrating content from multiple sources, including auto-generated API references and manually authored guides.
Documentation Assembly
The generateDocsDir.ts script is the central assembly piece. It performs the following tasks:
- Discovery: Uses
git ls-filesto find.mdfiles in the repository. - Transformation: Injects frontmatter (slugs, titles) and rewrites relative links for the hosted site.
- Inlining: Processes
{{ inline ... }}and{{ command-output ... }}directives to embed code snippets and CLI outputs directly into documentation.
Sources: docs-website/AGENTS.md:18-32, docs-website/README.md:84-118
Build Chain
AI and Agent Integration
DataHub provides an "Agent Context Kit" to bring metadata context to AI assistants and LLMs. This is facilitated through the Model Context Protocol (MCP) server, which allows tools like Claude, Cursor, and LangChain to query the DataHub context graph.
Key AI Components
- MCP Server: Connects AI assistants directly to DataHub metadata.
- DataHub Skills: Pre-built skills for AI agents to interact with data assets.
- Analytics Agent: Provides conversational analytics over data assets.
Sources: docs-website/static/llms.txt:18-28, docs-website/AGENTS.md:25-28
Summary
The DataHub architecture represents a complete end-to-end solution for metadata management. It combines a robust Python-based ingestion framework for data collection, a central Java-based Metadata Serving Tier for storage and retrieval, and a React-based frontend for user interaction. Its design prioritizes extensibility through the EntityRegistry and event-driven metadata changes via MCP/MCL, ensuring the platform can adapt to the evolving needs of modern data ecosystems.