Microbatch Execution Strategy
Relevant source files
The following files were used as context for generating this wiki page:
- crates/dbt-tasks-sa/src/runnable/microbatch.rs
- crates/dbt-parser/src/resolver.rs
- crates/dbt-schemas/src/schemas/nodes.rs
- crates/dbt-loader/src/dbt_macro_assets/dbt-fabricspark/macros/materializations/models/incremental/strategies.sql
- crates/dbt-parser/src/tests.rs
- crates/dbt-tasks-sa/src/microbatch/mod.rs
The Microbatch Execution Strategy is a specialized incremental materialization approach designed to process data in time-based windows or "batches." This strategy enhances traditional incremental models by providing granular control over data processing intervals, allowing for more efficient resource utilization and easier recovery from failures within specific time ranges.
In the dbt-core Rust implementation, microbatch execution involves significant context augmentation during the compilation and run phases. The system replaces standard macro functions with microbatch-aware variants that automatically inject event-time filters into upstream dependencies, ensuring that each batch only processes relevant data.
Sources: crates/dbt-tasks-sa/src/runnable/microbatch.rs:1-12, crates/dbt-schemas/src/schemas/nodes.rs:432-446
Architecture and Components
The microbatch system relies on the coordination between model configuration, batch context management, and Jinja environment extensions.
Core Data Structures
The strategy utilizes several key structures to manage the time-aware execution state:
| Component | Description |
|---|---|
BatchContext | Contains metadata for a specific execution window, including start/end times and batch ID. |
MicrobatchRefContext | Manages the time range and event-time mappings used by overridden ref and source functions. |
DbtIncrementalStrategy::Microbatch | The enum variant signaling that a model should be executed using this strategy. |
event_time | A critical configuration field for models and sources that defines the column used for time-based filtering. |
Sources: crates/dbt-tasks-sa/src/runnable/microbatch.rs:36-40, crates/dbt-schemas/src/schemas/nodes.rs:432-446
Execution Flow
The following diagram illustrates how a microbatch model is processed and how its context is extended for batch-specific rendering.
The execution engine iterates through batches, re-extending the context for every individual time window to ensure proper filtering.
Sources: crates/dbt-tasks-sa/src/runnable/microbatch.rs:31-155
Context Extension Logic
The function extend_microbatch_node_context is the primary entry point for preparing the Jinja environment for a microbatch run. It modifies the execution scope in several critical ways:
1. Macro Overrides
The standard ref and source functions are replaced with versions that carry a MicrobatchRefContext. When these functions are called within a microbatch model's SQL, they don't just return the relation name; they wrap the relation in a subquery that filters based on the event_time mapping of the upstream node and the current batch's time boundaries.
Sources: crates/dbt-tasks-sa/src/runnable/microbatch.rs:41-76
2. Batch Metadata Injection
Several internal variables are injected into the context to allow macro authors and adapter materializations to access batch state:
__dbt_microbatch_batch_id__: The unique identifier for the current batch.__dbt_microbatch_event_time_start__: Formatted SQL literal for the batch start.__dbt_microbatch_event_time_end__: Formatted SQL literal for the batch end.model.batch: A dictionary containingevent_time_startandevent_time_endasPyDateTimeobjects, enabling the use of Python-style datetime methods like.strftime()in Jinja.
Sources: crates/dbt-tasks-sa/src/runnable/microbatch.rs:85-135
3. Hook Control
To maintain consistency with dbt-core, microbatch execution restricts hook behavior:
- Pre-hooks: Injected into the context only if
batch_ctx.is_first()is true. - Post-hooks: Injected into the context only if
batch_ctx.is_last()is true.
Sources: crates/dbt-tasks-sa/src/runnable/microbatch.rs:188-195
Validation and Constraints
Event Time Requirements
For a microbatch model to filter data correctly, its upstream inputs (refs or sources) must have an event_time configuration. If a microbatch model depends on a node without an event_time, the system emits a MicrobatchModelNoEventTimeInputs warning, as no filtering can be applied, potentially leading to duplicate records.
Sources: crates/dbt-parser/src/resolver.rs:432-446
Adapter Support
Not all adapters support concurrent batch execution. If a model is configured with concurrent_batches: true on an adapter that does not support it, the system emits a warning and falls back to sequential batch execution.
Sources: crates/dbt-schemas/src/schemas/nodes.rs:1044-1065
SQL Generation
The SQL generation process for microbatches involves a specific rendering path that accounts for ephemeral models and batch-local filtering.
Incremental Strategy Logic
Adapters like dbt-fabricspark implement microbatch by wrapping insert_overwrite logic. The materialization requires a partition_by configuration that matches the grain of the batch_size.
Sources: crates/dbt-loader/src/dbt_macro_assets/dbt-fabricspark/macros/materializations/models/incremental/strategies.sql:63-75
Summary
The Microbatch Execution Strategy provides a robust mechanism for processing time-series data incrementally. By intercepting relation resolution (ref/source) and injecting time-bound window metadata, the strategy ensures that models remain idempotent and performant. Its integration into the Rust-based execution engine allows for strict validation of event_time dependencies and adapter-specific capabilities before execution begins.