Parquet Metadata Index
Relevant source files
The following files were used as context for generating this wiki page:
The Parquet Metadata Index is a core architectural component of dbt Core v2.0 (the Fusion engine) designed to replace the monolithic manifest.json with a high-performance, scalable, and queryable metadata storage system. By utilizing Apache Parquet as the primary artifact format, the system enables dramatically faster parse and compile times, especially for large projects, and allows for efficient, partial updates to the project state.
The system is split into two primary layers: raw metadata produced during the parse/compile phases (located in target/metadata/) and a consolidated, indexed snapshot (located in target/index/). This indexed data powers the interactive documentation server and serves as the source of truth for lineage, execution history, and project catalog information without requiring a live connection to a data warehouse.
Sources: crates/dbt-docs-server/README.md, README.md
System Architecture and Data Flow
The metadata lifecycle follows a multi-stage pipeline where raw parse results are first persisted as epoch-based Parquet files and subsequently ingested into a unified index.
The diagram above illustrates the transition from high-frequency raw metadata updates to the consolidated analytical index used for documentation and system inspection.
Sources: crates/dbt-docs-server/README.md, crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:1-20
Raw Metadata Storage (target/metadata/)
During the dbt parse phase, individual node updates are written to epoch files. This "Epoch Append" strategy ensures that only changed components are written to disk, avoiding the overhead of serializing the entire project state.
nodes/v1_{N}.parquet: Contains the serialized payload of models, seeds, and snapshots.alive.parquet: A snapshot of all currently activeunique_ids, used to prune deleted nodes.generation.parquet: A cold-start sentinel containing project-level metadata and hashes.
Sources: crates/dbt-metadata/src/parse_state.rs:1-35
Index Components and Table Schema
The consolidated index in target/index/ organizes metadata into specialized tables optimized for different access patterns (e.g., search, lineage, execution status).
Core Metadata Tables
| Table Name | Description | Key Fields |
|---|---|---|
nodes | Primary model/seed/source metadata. | unique_id, name, resource_type, materialization |
edges | Relationship graph (lineage). | parent_unique_id, child_unique_id, edge_type |
node_columns | Schema information for all nodes. | unique_id, column_name, data_type, description |
column_lineage | Fine-grained column-level dependencies. | from_node_unique_id, to_node_unique_id, kind |
| Sources: crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:430-580, crates/dbt-index-core/src/parquet.rs:10-50 |
Runtime and Execution Tables
| Table Name | Description | Key Fields |
|---|---|---|
invocations | History of dbt command executions. | invocation_id, command, generated_at |
run_results | Outcomes of node executions. | unique_id, invocation_id, status, execution_time |
source_freshness | Metadata for source data checks. | unique_id, max_loaded_at, status |
| Sources: crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:950-1080 |
Ingestion Logic and Merging
The ingestion engine converts raw metadata epochs into the consolidated index using two primary paths: Cold Ingest and Delta Ingest.
Delta Ingestion Flow
Delta ingestion allows dbt to update the index incrementally by processing only new epochs since the last run.
This flow minimizes I/O by utilizing the IngestState to track the high-water mark of processed data.
Sources: crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:156-230
Compaction Strategy
To prevent an accumulation of too many small Parquet files, a compaction trigger is built into the indexing logic. When the delta row count exceeds 10% of total alive nodes, or the file count exceeds a threshold (e.g., 8 files), the system merges all epochs into a single 0.parquet file and deletes the deltas.
Sources: crates/dbt-metadata/src/parse_state.rs:453-470, crates/dbt-schema-store/src/parquet_cache.rs:18-35
Key Data Structures
IngestState
The IngestState is the critical runtime structure used to synchronize the raw metadata with the consolidated index.
pub struct IngestState {
pub alive_mtime: Option<SystemTime>,
pub last_epoch: HashMap<String, u32>,
pub index_dir: Option<PathBuf>,
pub alive_ids: HashSet<String>,
}
Sources: crates/dbt-index-core/src/ingest/ingest_state.rs (referenced in crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:100-140)
PersistedState
Stored as .fusion_state.json in the index directory, this allows the ingest process to remain stateless between invocations.
Sources: crates/dbt-index-core/src/ingest/metadata_to_parquet.rs:55-75
Parquet-Backed Schema Cache
Separate from the general project metadata, dbt maintains a ParquetSchemaCache for column definitions. This cache handles:
- Build-hash guard: Rows containing a mismatched
CARGO_PKG_VERSIONare invalidated. - TTL eviction: Entries older than a configured interval (e.g., for warehouse-fetched schemas) are dropped.
- Lazy Deserialization: Schemas are kept as raw Arrow IPC bytes and only deserialized into memory when accessed, reducing memory pressure.
Sources: crates/dbt-schema-store/src/parquet_cache.rs:10-100
Summary
The Parquet Metadata Index represents a shift toward a decoupled, high-performance metadata layer in dbt. By using an epoch-based storage system for raw results and a consolidated Parquet index for documentation and analytical consumption, dbt Core v2.0 achieves significant performance gains. The combination of Arrow-based ingestion and specialized compaction strategies ensures that the system scales linearly with project size while maintaining sub-second metadata update capabilities.