Data Lineage
Relevant source files
The following files were used as context for generating this wiki page:
- metadata-ingestion/scripts/docgen.py
- e2e-test/ui/playwright/pages/lineage-base.page.ts
- metadata-ingestion/adding-source.md
- docs-website/static/llms.txt
- datahub-agent-context/src/datahub_agent_context/snowflake/generators/cortex_agent.py
- e2e-test/ui/playwright/tests/lineage-v3/v3-download-lineage-results.spec.ts
- datahub-web-react/CLAUDE.md
Data Lineage in DataHub provides a comprehensive, end-to-end visualization of the data supply chain. It connects diverse entities such as Datasets, DataJobs, DataFlows, Dashboards, and Charts into a unified context graph. This system enables users to understand technical dependencies, perform impact analysis, and track data transformations across the entire organization.
Sources: docs-website/static/llms.txt:5-9, metadata-ingestion/scripts/docgen.py:734-738
Lineage Architecture and Connections
DataHub supports multiple types of lineage connections, facilitating visibility from raw data ingestion to high-level business analytics. The lineage graph is built through both automatic extraction from source systems and manual or API-driven ingestion.
Supported Connection Types
The following table outlines the primary lineage relationships supported by the platform:
| Connection Type | Description |
|---|---|
| Dataset to Dataset | Direct transformation dependencies between tables or files. |
| DataJob to DataFlow | Mapping of individual tasks to their parent orchestration workflows. |
| DataJob to Dataset | Linking processing tasks to the datasets they read from or write to. |
| Chart to Dashboard | Association between individual visualizations and their parent dashboards. |
| Chart to Dataset | Linking visualizations to their underlying data sources. |
Sources: metadata-ingestion/scripts/docgen.py:743-747
Extraction Mechanisms
DataHub employs three primary strategies for populating lineage:
- Native Automatic Extraction: Supported for modern data tools like BigQuery, Snowflake, and dbt.
- SQL Parser: An advanced parser that analyzes database query logs to detect lineage with 97-99% accuracy for systems with limited native tracking.
- Manual/SDK: Programmatic lineage definition using the Python or Java SDKs for custom integrations.
Sources: metadata-ingestion/scripts/docgen.py:734-741
UI Implementation and ReactFlow
The Lineage UI is built using the ReactFlow library, which allows for dynamic, interactive graph visualizations. The implementation utilizes specific DOM identifiers to manage nodes, edges, and user interactions.
Component Structure
- Nodes: Represent individual entities. They use the ID format
lineage-node-{urn}orrf__node-{urn}. - Edges: Represent relationships. They use the format
rf__edge-{upstreamUrn}-:-{downstreamUrn}. - Column-Level Lineage (CLL): Visualizes dependencies at the field level. Column edges are identified by
rf__edge-{urn1}::{col1}-{urn2}::{col2}.
Sources: e2e-test/ui/playwright/pages/lineage-base.page.ts:1-15
User Interactions Flow
The following sequence diagram illustrates the process of a user interacting with a lineage node to expand its dependencies:
The diagram shows the synchronous flow from a UI click to the expansion of dependencies in the ReactFlow canvas.
Sources: e2e-test/ui/playwright/pages/lineage-base.page.ts:133-145, datahub-web-react/CLAUDE.md:46-52
Impact Analysis
Impact Analysis is a core feature of the lineage system that allows users to determine the "downstream" consequences of changes to a "upstream" data asset.
Degree of Dependency
DataHub categorizes dependencies by their distance from the source entity:
- 1st Degree: Direct consumers or producers.
- 2nd Degree: Indirect dependencies (neighbors of neighbors).
- 3+ Degree: Deep dependencies across the supply chain.
Users can filter the Impact Analysis view by these degrees and export the results to CSV for external reporting.
Sources: e2e-test/ui/playwright/tests/lineage-v3/v3-download-lineage-results.spec.ts:15-30, e2e-test/ui/playwright/pages/lineage-base.page.ts:71-78
Visual Representation of Lineage Flow
This flowchart represents a typical end-to-end lineage path from an ingestion source (Kafka) through processing (Airflow/Snowflake) to consumption (Looker).
Sources: metadata-ingestion/scripts/docgen.py:743-747, docs-website/static/llms.txt:47-49
Developer and Agent Integration
For technical users and AI agents, DataHub provides programmatic access to lineage metadata via structured APIs and tools.
Agent Context Tools
The Agent Context Kit enables AI agents (like Snowflake Cortex) to interact with the lineage graph using specific functions:
| Tool Name | Purpose | Input Parameters |
|---|---|---|
get_lineage | Retrieves upstream or downstream graph. | urn, column_name, upstream, max_hops |
get_lineage_paths_between | Shows detailed transformation paths between two entities. | source_urn, target_urn, source_column, target_column |
Sources: datahub-agent-context/src/datahub_agent_context/snowflake/generators/cortex_agent.py:321-356
Ingestion Configuration
When developing a new ingestion source, lineage capabilities are declared using Python decorators. This ensures the source is correctly categorized in the DataHub UI and documentation.
@capability(
SourceCapability.LINEAGE_COARSE,
"Table-level lineage supported via query log analysis",
supported=True
)
@capability(
SourceCapability.LINEAGE_FINE,
"Column-level lineage supported for select dialects",
supported=True
)
Sources: metadata-ingestion/adding-source.md:94-118, metadata-ingestion/scripts/docgen.py:840-850
Conclusion
Data Lineage serves as a foundational component of DataHub's metadata graph, providing the necessary visibility for data governance, quality, and observability. By integrating UI-based visualization, impact analysis, and programmatic tools for AI agents, it ensures that both human teams and automated systems can navigate the complexities of modern data stacks.