Time Machine & Record Replay
Relevant source files
The following files were used as context for generating this wiki page:
The Time Machine & Record Replay system is a set of capabilities within dbt Core v2.0 designed to improve performance at scale and enable isolated testing. It leverages parquet-based artifacts to "time travel" through project states, replaying previous events and executions without requiring a live connection to a data warehouse. This system is foundational to the Fusion engine, providing a read-only, stateless environment that loads project snapshots into an in-memory analytical engine.
At its core, the system distinguishes between Partial-Parse Replay, which memoizes compilation results to skip redundant WalkDir scans, and Event Record/Replay, which captures live SQL execution results into "tome" files for deterministic differential testing.
Sources: crates/dbt-docs-server/README.md, crates/dbt-metadata/src/partial_parse.rs:7-14
Incremental Parse and State Replay
The Time Machine's "Partial Parse" functionality acts as a cache that memoizes the resolution of project files. The cache key is composed of (file_path, mtime) pairs for every tracked file. This allows the system to return the same nodes as a full parse would, but in a fraction of the time.
The Three Execution Outcomes
When a project is loaded with --partial-parse, the engine evaluates the current state against the cached IncrementalState to determine one of three paths:
| Outcome | Trigger | Mechanism | Approximate Time |
|---|---|---|---|
| Fast-path | No files changed + --partial-load | Zero WalkDir; returns the previous ResolverState as-is from parquet. | ~10ms |
| Incremental | Model/Analysis .sql changed | Only changed files are re-resolved; unchanged nodes are replayed from the cache. | ~500ms |
| Full Parse | Non-incremental file change (YAML, Macros) | Cache is discarded; all files are re-resolved from scratch. | ~1.8s |
Sources: crates/dbt-metadata/src/partial_parse.rs:18-36, crates/dbt-metadata/src/partial_parse.rs:595-620
Mermaid: Partial Parse Logic Flow
This diagram illustrates how the system decides whether to replay a previous state or trigger a fresh parse.
The system uses blake3 hashes of dbt_project.yml, profiles.yml, and CLI --vars to ensure the replayed state remains semantically identical to a full parse.
Sources: crates/dbt-metadata/src/partial_parse.rs:40-52, crates/dbt-metadata/src/partial_parse.rs:125-140
Record & Replay in Compatibility Testing
The system implements a specialized Record/Replay mode for dialect compatibility (e.g., Snowflake, BigQuery). This allows developers to verify the SQL compiler's behavior against pre-recorded "ground truth" results stored in Parquet "tomes."
Operating Modes
- Record Mode: Executes compiled SQL on a live warehouse cluster (e.g., Snowflake) and captures the resulting
RecordBatchstreams into a parquet tome. - Replay Mode: Loads pre-recorded results from
tests/data/recordings.parquet. This mode requires no live warehouse connection, making CI/CD pipelines faster and more reliable.
Sources: crates/dbt-metricflow/tests/snowflake_compat.rs:1-17, crates/dbt-metricflow/tests/snowflake_compat.rs:142-156
Data Normalization for Replay
To ensure that replayed events are deterministic and comparable, the system applies normalization to recorded data. This involves stripping non-deterministic OS data from logs and error messages.
- Thread IDs: OS thread IDs in Rust panic messages (e.g.,
(12345678)) are replaced with stable placeholders. - Replay Paths: Absolute recording paths are stripped so that replayed tests are not tied to a specific local machine or username.
- Timestamps: ISO 8601 timestamps and durations are normalized to stable tokens (e.g.,
[duration]).
Sources: crates/dbt-test-utils/src/task/utils.rs:242-255, crates/dbt-test-utils/src/task/utils.rs:188-210
Event Formatting and Status Replay
When replaying node execution events, the system must reconstruct the status and visual feedback for the user interface. The NodeProcessed and NodeEvaluated events capture the final state of a node, which is then formatted using standardized color and padding rules.
Node Outcome Statuses
Replayed events use a mapping to translate raw outcomes into human-readable statuses:
| Outcome | Context | UI Status | Color |
|---|---|---|---|
Success | Standard Node | success | Green |
Success | Test Passed | pass | Green |
Success | Node with Warnings | warn | Yellow |
Skipped | Upstream Failure | skipped | Yellow |
Skipped | Build Cache Reuse | reused | Green |
Error | Any Phase | error/fail | Red |
Sources: crates/dbt-common/src/tracing/formatters/node.rs:163-195, crates/dbt-common/src/tracing/formatters/node.rs:202-230
Mermaid: Event Replay Sequence
This diagram shows how a recorded event is rehydrated into a UI status during a replayed session.
For cached or replayed nodes, the duration is forced to zero to indicate that no actual warehouse work was performed during the current session.
Sources: crates/dbt-common/src/tracing/formatters/node.rs:260-275, crates/dbt-common/src/tracing/formatters/node.rs:290-305
Conclusion
The Time Machine & Record Replay module transforms dbt Core from a reactive CLI into a state-aware analytical engine. By combining IncrementalState validation with Parquet-based event recording, the project achieves significant performance gains (reducing cold parse times from ~7.5s to ~10ms replays) and provides a robust framework for cross-dialect compatibility testing without the overhead of live warehouse infrastructure.
Sources: crates/dbt-metadata/src/partial_parse.rs:56-65, crates/dbt-metricflow/tests/metricflow_compat.rs:11-20