Data Lineage Tracking
Relevant source files
The following files were used as context for generating this wiki page:
Data Lineage Tracking within dbt-core (v2/Fusion) is a multi-layered system designed to capture, persist, and visualize the relationships between various data assets. It operates at two primary grains: project-level (node-to-node) and column-level (CLL). The system utilizes a Rust-based engine to parse SQL and Python models, extracting dependencies through static analysis to build a Directed Acyclic Graph (DAG) that represents the flow of data from sources to final analytics models.
The lineage information is persisted as Parquet artifacts, which are high-performance analytical files written during dbt commands like compile, run, or build. These artifacts are then ingested by an analytical engine to power interactive documentation and IDE features, such as the lineage panel in VS Code. This architecture allows for stateless, read-only documentation serving without a live warehouse dependency.
Sources: crates/dbt-docs-server/README.md, crates/dbt-features/src/index.rs:170-200, crates/dbt-parser/src/resolve/resolve_models.rs
Lineage Architecture and Grains
The system tracks lineage at different levels of granularity, referred to as "grains." Users can switch between these views depending on the depth of analysis required.
Project-Level Lineage
Project-level lineage represents the dependencies between dbt resources, such as models, sources, seeds, and snapshots. This forms the standard dbt DAG where nodes represent entire tables or views.
Column-Level Lineage (CLL)
Column-level lineage tracks how specific data elements move through the system. It identifies the upstream columns that contribute to a specific downstream column, including transformations. CLL edges are represented by the CllEdge structure, which maps a "from" node and column to a "to" node and column.
Sources: crates/dbt-lineage-core/src/lib.rs, crates/proto-rust/include/dbtlabs/proto/public/v1/fields/vscode_types.proto:145-155
Data Flow Overview
The following diagram illustrates how lineage data is extracted from the project and flows into the visualization layers.
The flow starts with the parsing of dbt models, followed by dependency extraction, persistence into metadata artifacts, and finally serving through an analytical index.
Sources: crates/dbt-docs-server/README.md, crates/dbt-features/src/index.rs:114-150
Metadata Persistence and Ingestion
Lineage information is written to specific subdirectories within the target/metadata/ directory during the execution of the Fusion engine.
Parquet Artifacts
The lineage tracking system relies on epoch-based Parquet files for high-performance ingestion. The primary directories involved include:
compile/column_lineage: ContainsCllRowentries defining edges between columns.compile/nodes: Stores compiled code, hashes, and grain information (declared, tested, and inferred).parse/nodes: Contains the base project structure anddepends_onrelationships.
Sources: crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:25-50, crates/dbt-features/src/index.rs:340-380
Ingestion Logic
The metadata_to_parquet module provides a high-performance path for moving data from temporary metadata epochs into a permanent index. It uses a "Carry Forward Merge" strategy to ensure that compiled fields (like CLL) are preserved even when a partial re-parse occurs.
| Subdirectory | Key Column | Carry Columns |
|---|---|---|
column_lineage | child_unique_id | N/A (Replace/Prune) |
nodes | unique_id | compiled_code, grain, table_role |
Sources: crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:520-560
Implementation Components
Lineage Interaction Types
User interactions with lineage visualizations are tracked via a standardized set of events, allowing the system to respond to refocusing, grain switches, and navigation.
| Interaction Type | Description |
|---|---|
SELECT_NODE | Single-click interaction to view node details. |
SWITCH_TO_COLUMN_LINEAGE | Changes the view from project nodes to specific columns. |
SHOW_UPSTREAM | Filters the graph to show only parent dependencies. |
REFOCUS_ON_NODE | Centers the lineage graph on a specific asset. |
Sources: crates/proto-rust/include/dbtlabs/proto/public/v1/fields/vscode_types.proto:110-140
Core Data Structures
Lineage logic is underpinned by Rust structures that define the edges and node information.
// Representative structure for Column Level Lineage edges
pub struct CllEdge {
pub from_node: String,
pub from_col: String,
pub to_node: String,
pub to_col: String,
pub op: LineageKind, // e.g., Transform, Copy
}
Sources: crates/dbt-lineage-core/src/lib.rs, crates/dbt-features/src/index.rs:15-25
Static Analysis and Extraction
Dependency extraction occurs during the "Resolve" phase. For SQL models, the system parses ref() and source() calls. For Python models, it uses AST analysis to identify these calls without executing the Python code.
Reference Parsing Logic
The parser identifies dependencies in model configurations and constraints using specialized regex or AST walkers. For example, the parse_ref_from_constraint function extracts the package, model name, and version from strings like ref('pkg', 'model', version=2).
Sources: crates/dbt-parser/src/resolve/resolve_models.rs:95-135
This sequence demonstrates the extraction of dependencies during the parsing phase to build the internal representation of the lineage graph.
Sources: crates/dbt-parser/src/resolve/resolve_models.rs:820-850
Conclusion
Data Lineage Tracking in this project is a sophisticated pipeline that bridges static code analysis with high-performance analytical storage. By decoupling the capture of lineage (during compilation) from its consumption (in the docs server or IDE), dbt-core provides a responsive and scalable way to understand data flows. The system's ability to track data at both the resource and column levels ensures comprehensive visibility into the data transformation lifecycle.