Ingestion Sources & Connectors
Relevant source files
The following files were used as context for generating this wiki page:
Ingestion Sources are the components within the DataHub framework responsible for extracting metadata from external data systems. These sources pull information from various platforms—such as Snowflake, BigQuery, or Kafka—and transform them into a standardized format for consumption by DataHub.
The framework is heavily inspired by Apache Gobblin, utilizing a standardized "MetadataChangeEvent" (MCE) or "MetadataChangeProposal" (MCP) to facilitate the flow of data from diverse sources into a central sink. Sources are typically configured via YAML-based "Recipes" that define the source type, configuration parameters, and the target sink.
Sources: metadata-ingestion/source_overview.md:5-9, metadata-ingestion/developing.md:120-130, metadata-ingestion/recipe_overview.md:5-10
Core Architecture
The ingestion framework operates on a modular producer-consumer model. A Source produces metadata events, which are wrapped in MetadataWorkUnits, and a Sink consumes these events to persist them in DataHub.
The diagram above shows the standard flow of metadata from an external system through the ingestion pipeline to the DataHub backend.
Sources: metadata-ingestion/developing.md:120-130, metadata-ingestion/src/datahub/ingestion/source/rdf/ingestion/README.md:7-13
Key Components
| Component | Description |
|---|---|
| Source | The plugin that extracts metadata from a specific system. |
| ConfigModel | A Pydantic-based class that defines and validates configuration parameters for the source. |
| SourceReport | A reporter interface used to track statistics, warnings, and failures during an ingestion run. |
| MetadataWorkUnit | A wrapper for metadata events (MCE/MCP) that includes additional context for the ingestion pipeline. |
| Recipe | A YAML configuration file specifying the source, sink, and any necessary transformers. |
Sources: metadata-ingestion/adding-source.md:23-45, metadata-ingestion/recipe_overview.md:5-15, metadata-ingestion/src/datahub/ingestion/source/rdf/ingestion/README.md:15-25
Connector Implementation
Building a new metadata ingestion source requires extending the Python ingestion framework. The core logic resides in the get_workunits_internal method of the Source class.
1. Configuration Handling
DataHub uses Pydantic for configuration management. All configuration models must inherit from ConfigModel. Developers are encouraged to match the terminology of the source system (e.g., using account_id for Snowflake instead of a generic host_port).
Sources: metadata-ingestion/adding-source.md:23-26, metadata-ingestion/CLAUDE.md:105-110
from datahub.api.configuration.common import ConfigModel
from pydantic import Field
class CustomSourceConfig(ConfigModel):
api_key: str = Field(description="API Key for the source system")
database: str = Field(description="Database name to ingest")
Sources: metadata-ingestion/adding-source.md:28-36
2. The Source Logic
The source must implement the get_workunits_internal method (or get_workunits), which yields a stream of MetadataWorkUnit objects. These objects typically contain MCPs (Metadata Change Proposals) generated from the source system's API or metadata logs.
Sources: metadata-ingestion/adding-source.md:43-47, metadata-ingestion/src/datahub/ingestion/source/rdf/ingestion/README.md:30-40
The sequence above illustrates the interaction between the ingestion pipeline and a source plugin during a metadata extraction run.
Sources: metadata-ingestion/src/datahub/ingestion/source/rdf/ingestion/README.md:30-40, metadata-ingestion/adding-source.md:38-41
Metadata Source Status & Capabilities
DataHub uses decorators to declare metadata about the connector itself, which are then used by the docGen system to generate documentation.
Support Status
Connectors are categorized into three levels of reliability:
- Certified: Well-tested and widely adopted.
- Incubating: Ready for adoption but potentially missing edge-case handling.
- Testing: Experimental and subject to change.
Sources: metadata-ingestion/source_overview.md:20-30, metadata-ingestion/adding-source.md:76-80
Capabilities
Connectors use the @capability decorator to indicate supported features, such as:
DELETION_DETECTION(Stateful Ingestion)DOMAINSDATA_PROFILINGLINEAGE_COARSE/LINEAGE_FINEPLATFORM_INSTANCE
Sources: metadata-ingestion/adding-source.md:81-85, metadata-ingestion/scripts/docgen.py:60-70
Discoverability and Registry
To be discoverable by the DataHub CLI (e.g., datahub check plugins), the source must be registered as an entry point in setup.py (or pyproject.toml). This sets up a shortened alias for use in recipes.
Sources: metadata-ingestion/adding-source.md:59-62, metadata-ingestion/CLAUDE.md:120-130
# In setup.py
entry_points={
"datahub.ingestion.source.plugins": [
"custom-source = datahub.ingestion.source.custom:CustomSource",
],
}
Sources: metadata-ingestion/adding-source.md:59-62, metadata-ingestion/src/datahub/ingestion/source/rdf/ingestion/README.md:55-60
Recipes and Execution
A Recipe is the main configuration file for ingestion, defining exactly one source and one sink.
source:
type: mssql
config:
username: sa
password: ${MSSQL_PASSWORD}
database: DemoData
sink:
type: "datahub-rest"
config:
server: "http://localhost:8080"
Sources: metadata-ingestion/recipe_overview.md:15-30
Ingestion Execution Methods
- CLI: Run via
datahub ingest -c recipe.yaml. - UI: Managed Ingestion via the DataHub frontend, allowing execution without local CLI setup.
Sources: metadata-ingestion/recipe_overview.md:40-60
Summary
The Ingestion Sources & Connectors framework provides a standardized, extensible architecture for pulling metadata into DataHub. By leveraging Pydantic for validation, standardizing on the MCE/MCP format, and using a registry-based plugin system, DataHub maintains a broad ecosystem of connectors that are consistent and highly configurable.
Sources: metadata-ingestion/developing.md:120-130, metadata-ingestion/source_overview.md:5-10