Agate Reimplementation (dbt-agate)
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The dbt-agate crate provides a Rust-native reimplementation of the core surface area of the Python agate library. It is specifically designed for the Fusion engine's MiniJinja layer, providing a typed, immutable row-oriented data abstraction that integrates directly into the Jinja execution context. AgateTable instances serve as first-class values in Jinja, enabling developers to reference and transform data sets within dbt models or operations using familiar syntax like {{ }} and {% %}.
The primary purpose of this reimplementation is to bridge the gap between Arrow's high-performance columnar data format and the row-oriented expectations of Jinja templates. Fusion operates natively on RecordBatch data, and dbt-agate ensures that this data can be transformed (e.g., via group_by()) and iterated upon row-by-row without sacrificing the efficiency of the underlying Arrow representation.
Sources: crates/dbt-agate/ARCHITECTURE.md:3-9
Core Architecture
The architecture follows a "columnar-first, row-when-necessary" philosophy. Data flows from upstream sources as Arrow RecordBatch objects and is transformed into a lazy, row-wise representation only when required by Jinja iteration or specific row-level operations.
Data Flow Pipeline
- Arrow
RecordBatch: Raw columnar data arrives from storage engines. FlatRecordBatch: Structural normalization occurs here. Nested Arrow columns (structs, lists) are flattened into a flat schema, and Arrow types are mapped to Agate types.TableRepr: The authoritative representation of table structure. It owns theFlatRecordBatchand manages an optional, lazyVecOfRows.AgateTable: The public façade that implementsminijinja::Object. It triggers lazy row materialization and cell conversion intominijinja::Valueobjects on demand.
Sources: crates/dbt-agate/ARCHITECTURE.md:20-30
The following diagram illustrates the transformation from raw Arrow data to Jinja-compatible values:
Key Components and Abstractions
The reimplementation relies on several cooperating abstractions to emulate Python Agate behavior within the Rust/Jinja environment.
| Component | Description |
|---|---|
AgateTable | The primary object exposed to Jinja. Wraps TableRepr and handles method dispatch. |
TableRepr | Manages the schema, row counts, and access semantics. Owns the data buffers. |
MappedSequence | A protocol defining Python-sequence semantics (indexing, iteration) for rows and columns. |
Tuple / TupleRepr | Virtualized tuple-like behavior enabling lazy materialization of row data. |
ArrayConverter | Responsible for the "Arrow -> Jinja" bridge, transforming Arrow cells into scalar values. |
Sources: crates/dbt-agate/ARCHITECTURE.md:38-60
Converters and Type Mapping
Converters are critical for maintaining cross-platform conformance. They must honor the full Arrow spec, including nullability, numeric width, and decimal precision/scale. The system preserves null correctness by respecting the Arrow null buffer independently of the value buffer, ensuring that Arrow nulls become None in Jinja.
Sources: crates/dbt-agate/ARCHITECTURE.md:65-75
Usage in dbt Materializations
Agate tables are extensively used in dbt macros, particularly for loading seed data and managing materializations. In the Databricks adapter, for example, the agate_table is used to infer types and batch rows for SQL insertion.
{% macro databricks__load_csv_rows(model, agate_table) %}
{% set batch_size = get_batch_size() %}
{% for chunk in agate_table.rows | batch(batch_size) %}
{% set sql %}
insert into {{ this.render() }} values
{% for row in chunk -%}
({%- for col_name in agate_table.column_names -%}
{%- set inferred_type = adapter.convert_type(agate_table, loop.index0) -%}
cast({{ get_binding_char() }} as {{inferred_type}})
{%- endfor -%})
{%- endfor %}
{% endset %}
{% do adapter.add_query(sql, bindings=bindings) %}
{% endfor %}
{% endmacro %}
Sources: crates/dbt-loader/src/dbt_macro_assets/dbt-databricks/macros/materializations/seeds/helpers.sql:5-30
Integration in the Resolve Phase
During the dbt resolution phase, AgateTable is used to handle data returned by run_query and to process seed configurations. For instance, resolve_sources and resolve_models interact with table properties and configurations that are often passed through the Agate abstraction layer to determine relation names and column types.
Sources: crates/dbt-parser/src/resolve/resolve_sources.rs:205-220
Internal Data Relationships
The AgateTable structure maintains a strict hierarchy to ensure data integrity while allowing for zero-copy operations wherever possible.
Conclusion
The dbt-agate reimplementation is a specialized performance layer for dbt v2. By treating Arrow as the authoritative data source and lazily materializing Jinja-compatible rows, it allows dbt to maintain high throughput during large-scale data processing while preserving the flexible, template-driven transformation logic required by dbt developers. This architecture ensures that scalar values used in templates are always consistent with the underlying optimized columnar data.