State Management & Caching
Relevant source files
The following files were used as context for generating this wiki page:
The state management and caching system in dbt-core v2.0 is designed to dramatically improve performance by memoizing the resolution of project source files. It operates on a conceptual model where the result of the project resolution phase—the ResolverState—is cached based on a key consisting of the file paths and modification times (mtimes) of all tracked files in the project. This allows for three primary execution outcomes: a zero-scan "fast-path" for unchanged projects, an "incremental" parse for SQL changes, and a "full parse" when structural changes occur.
Sources: crates/dbt-metadata/src/partial_parse.rs:5-18
The system primarily utilizes a pure-Rust, parquet-backed cache located in the target/parse_state/ directory. This cache is read-only and stateless from the perspective of the documentation server, loading parquet artifacts into an in-memory DuckDB engine at boot time to answer queries without requiring a live warehouse dependency.
Sources: crates/dbt-metadata/src/parse_state.rs:1-4, crates/dbt-docs-server/README.md:15-18
Cache Architecture and Storage
The caching layer uses a multi-file parquet structure to store project metadata, file timestamps, and node definitions. This approach allows for efficient epoch-based updates and filtered loading.
Directory Layout
The cache is organized within the project's target directory to maintain distinct versions of state and allow for periodic compaction.
The diagram shows the file structure of the parquet-backed incremental parse cache.
Sources: crates/dbt-metadata/src/parse_state.rs:6-15
Epoch Append Strategy
To avoid full rewrites on every change, the system uses an append-only strategy for nodes:
- Cold Start: Writes
nodes/0.parquetand clears existing epochs. - Incremental Update: Writes a new
nodes/N.parquetcontaining only changed rows. - Read Merge: Latest epoch wins by
unique_idduring deserialization. - Compaction: Triggered when the delta row count exceeds 10% of alive nodes or when more than 8 epoch files exist.
Sources: crates/dbt-metadata/src/parse_state.rs:19-27
Partial Parsing Logic
The partial parsing engine determines if a previous compilation can be reused by validating the environment and scanning for file changes.
Invalidation Triggers
Specific events trigger different parsing outcomes based on the risk to semantic integrity.
| Event | Detection Method | Outcome |
|---|---|---|
| No files changed | All file mtimes match | Fast-path (~10ms) |
| Model/Analysis SQL changed | File mtime changed | Incremental (~500ms) |
| YML/Config/Macro changed | mtime + extension check | Full Parse (~1.8s) |
| File deleted | Stat failure | Full Parse (~1.8s) |
dbt_project.yml changed | blake3 content hash | Full Parse (~1.8s) |
| Binary version changed | CARGO_PKG_VERSION mismatch | Full Parse (~1.8s) |
Sources: crates/dbt-metadata/src/partial_parse.rs:53-70
Validation Process
Before reusing state, the system performs a sequence of validation checks.
The sequence shows the validation steps required before a cached state is accepted for reuse.
Sources: crates/dbt-login/src/state_guidance.rs:35-61, crates/dbt-metadata/src/partial_parse.rs:125-160
Partial Loading and Selection
The --partial-load feature allows dbt to deserialize only a subset of nodes from the parquet cache, further reducing boot times for specific commands like seed or snapshot.
Selection Filtering
When a selector is used with --partial-load, the system attempts to resolve a unique_id filter using only the node_index (no payload deserialization). This is limited to "index-safe" methods.
- Safe Methods (Index-only):
fqn,path,file,tag,package,resource_type,source. - Unsafe Methods (Require Payload):
config,state,result,source_status,group,version.
Sources: crates/dbt-metadata/src/partial_parse.rs:556-574, crates/dbt-metadata/src/partial_parse.rs:595-605
Dependency Closure
A critical invariant of partial loading is the "liveness invariant." Every loaded node must have its dependencies present in the loaded set. If a node references a dependency not included in the filtered load, the system falls back to a full load to prevent scheduler errors.
Sources: crates/dbt-metadata/src/partial_parse.rs:258-270, crates/dbt-metadata/src/parse_state.rs:29-37
State Management Implementation
Configuration Detection
dbt checks for state management enablement through multiple sources in a specific priority order:
- Environment Variable:
DBT_ENGINE_MANAGE_STATE - Project File:
./dbt_project.yml - User Settings:
~/.dbt/user_settings.yml
Sources: crates/dbt-login/src/state_guidance.rs:35-61
Compilation Integration
The compilation lifecycle integrates state loading in its first phase.
The flowchart illustrates how the compilation process hydrates the cache and handles unchanged nodes.
Sources: crates/dbt-compilation/src/core.rs:62-137
Implementation Snippet: Cache Validation
pub fn validate(
&self,
cli_vars: &Option<BTreeMap<String, dbt_yaml::Value>>,
) -> Option<&'static str> {
if self.cli_vars_hash != hash_cli_vars(cli_vars) {
return Some("--vars changed");
}
if self.profile_hash != self.dbt_profile.blake3_hash() {
return Some("profile config changed");
}
// ... additional checks for profiles.yml and env vars
}
Sources: crates/dbt-metadata/src/partial_parse.rs:129-138
Conclusion
State Management in dbt-core v2.0 leverages a sophisticated parquet-backed caching strategy to minimize redundant parsing. By distinguishing between incremental-safe changes (SQL models/analyses) and structural changes (YML/Macros), the system balances high performance with semantic correctness. The integration of --partial-load and index-based selection further optimizes the developer workflow by reducing the memory and I/O overhead of large-scale project initialization.
Sources: crates/dbt-metadata/src/partial_parse.rs:5-43, crates/dbt-metadata/src/parse_state.rs:39-44