Metadata Ingestion Framework
Relevant source files
The following files were used as context for generating this wiki page:
The Metadata Ingestion Framework is a Python-based system designed to extract metadata from diverse data sources and transmit it to DataHub. It utilizes a modular architecture inspired by Apache Gobblin, where standardized metadata objects flow from producers (Sources) through optional processors (Transformers) to consumers (Sinks). This framework enables DataHub to serve as a unified context graph for discovery, governance, and observability.
The framework supports a wide array of integrations, including databases (BigQuery, Snowflake), orchestrators (Airflow, Dagster), and BI tools (Looker, Tableau). Developers can extend the framework by implementing custom connectors that adhere to the internal API patterns.
Sources: metadata-ingestion/developing.md:129-135, metadata-ingestion/source_overview.md:5-13, docs-website/static/llms.txt:5-9
Framework Architecture
The ingestion framework operates on a modular pipeline architecture. Metadata is extracted by a Source, optionally modified by one or more Transformers, and finally sent to a Sink.
Core Components
| Component | Description |
|---|---|
| Source | Extracts metadata from external systems (e.g., Snowflake, MySQL) and produces MetadataWorkUnit objects. |
| Transformer | Modifies metadata events in-flight, such as adding tags, ownership, or terms to datasets. |
| Sink | Consumes metadata events and writes them to a destination, typically the DataHub GMS via REST or Kafka. |
| Recipe | A YAML configuration file that defines the source, sink, and transformer settings for an ingestion run. |
Sources: metadata-ingestion/recipe_overview.md:5-15, metadata-ingestion/developing.md:129-135, metadata-ingestion/recipe_overview.md:81-85
Metadata Data Flow
The following diagram illustrates the high-level flow of metadata from an external data system through the DataHub ingestion pipeline.
Implementation Details
Sources and Work Units
The core of a source implementation is the get_workunits_internal method. This method produces a stream of metadata events, typically Metadata Change Proposal (MCP) objects, wrapped within a MetadataWorkUnit.
Sources: metadata-ingestion/adding-source.md:58-60
Configuration with Pydantic
The framework uses Pydantic for configuration management. All configuration models must inherit from ConfigModel. This ensures type safety, validation, and automatic documentation generation for ingestion recipes.
- Secret Handling: Sensitive fields like passwords or tokens use the
SecretStrtype to prevent accidental exposure in logs. - Filtering: Standardized
AllowDenyPatternsare used for inclusive or exclusive filtering of assets (e.g.,table_pattern).
Sources: metadata-ingestion/CLAUDE.md:103-112, metadata-ingestion/adding-source.md:32-35
Pipeline Execution
Ingestion runs are executed using the datahub ingest CLI command or via the DataHub UI. The execution lifecycle involves:
- Parsing: The Recipe YAML is parsed into Pydantic models.
- Initialization: The Source, Sink, and Transformers are instantiated with the provided configuration.
- Streaming: The Source yields
MetadataWorkUnitswhich are processed by Transformers and sent to the Sink. - Reporting: A
SourceReporttracks statistics, warnings, and failures during the run.
Sources: metadata-ingestion/recipe_overview.md:51-55, metadata-ingestion/adding-source.md:54-56, metadata-ingestion/src/datahub/ingestion/run/pipeline.py
Ingestion Sequence
The following sequence diagram represents the interaction between the Ingestion Runner and the framework components during a typical run.
Development Guidelines
Plugin Registration
New sources are registered as entry points in the setup.py file. This allows the DataHub CLI to discover the plugin and use its shortened alias in recipes.
# Example entry point registration in setup.py
entry_points={
"datahub.ingestion.source.plugins": [
"my-source = datahub.ingestion.source.my_source:MySourceClass",
],
}
Sources: metadata-ingestion/CLAUDE.md:126-133, metadata-ingestion/adding-source.md:76-78
Code Style and Standards
- Type Annotations: Mandatory for all new code; prefer
Protocolfor interfaces. - Naming: Must match the terminology of the source system (e.g.,
account_idfor Snowflake). - Testing: Tests are located in the
tests/directory and use thepytestframework. Integration tests often use Docker to simulate source systems.
Sources: metadata-ingestion/CLAUDE.md:65-75, metadata-ingestion/developing.md:144-148, metadata-ingestion/adding-source.md:80-82
Summary
The Metadata Ingestion Framework provides a robust, extensible foundation for synchronizing metadata from the entire data stack into DataHub. By enforcing strict configuration standards through Pydantic and modularizing the flow via Sources, Transformers, and Sinks, it ensures that metadata remains accurate, discoverable, and actionable across the organization.
Sources: metadata-ingestion/source_overview.md:5-13, metadata-ingestion/developing.md:129-135