MetricFlow Semantic Layer
Relevant source files
The following files were used as context for generating this wiki page:
The MetricFlow Semantic Layer within dbt provides a structured framework for defining business metrics and semantic models independently of the physical data warehouse schema. Its primary purpose is to allow data analysts to define logic once and expose it across various downstream tools through a unified semantic manifest. This layer handles the translation of high-level metric queries into dialect-specific SQL, such as DuckDB or Snowflake.
Sources: crates/dbt-metricflow/tests/metricflow_compat.rs:4-10, crates/dbt-parser/src/resolver.rs:596-610
Architecture and Components
The Semantic Layer is built upon a hierarchy of semantic objects defined in YAML and resolved into a SemanticManifest. This manifest serves as the single source of truth for the semantic query compiler.
Core Semantic Objects
The system relies on several key components to define the semantic meaning of data:
| Component | Description |
|---|---|
| Semantic Model | Maps a physical relation (table/view) to a semantic entity. Contains measures, dimensions, and entities. |
| Metric | A quantitative value calculated over a semantic model. Includes simple, derived, ratio, cumulative, and conversion types. |
| Dimension | Categorical or time-based attributes used for grouping and filtering (e.g., ds, country). |
| Entity | Joins keys that define relationships between semantic models (e.g., listing, user). |
| Saved Query | Predefined query templates containing metrics, group-bys, and filters. |
Sources: crates/dbt-metricflow/tests/metricflow_compat.rs:114-160, crates/dbt-schemas/src/schemas/semantic_layer/semantic_manifest.rs:14-19
Compilation Flow
The MetricFlow semantic query compiler translates high-level specifications into SQL. The process involves parsing a query spec, resolving it against the InMemoryMetricStore, and generating SQL for a specific dialect.
The diagram above represents the flow from a metric query request to executable SQL.
Sources: crates/dbt-metricflow/tests/metricflow_compat.rs:1120-1130, crates/dbt-metricflow/tests/data/convert_python_tests.py:46-52
Metric Types and Logic
The Semantic Layer supports complex analytical patterns through specialized metric types. These are defined in resolve_metrics.rs and consumed by the compiler.
Metric Classification
Metrics are classified based on their underlying calculation logic:
- Simple Metrics: Direct aggregations (sum, count, min, max, average) over a single measure in a semantic model.
- Derived Metrics: Mathematical expressions combining multiple other metrics (e.g.,
bookings - bookings_1_day_ago). - Ratio Metrics: A numerator metric divided by a denominator metric.
- Cumulative Metrics: Aggregations over a rolling window or grain-to-date (e.g., trailing 2 months revenue).
- Conversion Metrics: Measures the rate of successful transitions between two states (e.g., visits to buys) within a specific time window.
Sources: crates/dbt-metricflow/tests/metricflow_compat.rs:172-350, crates/dbt-parser/src/resolve/resolve_metrics.rs:374-420
Non-Additive Dimensions and Semi-Additive Measures
For snapshots (like account balances), the layer supports semi-additive patterns where measures are summed across some dimensions but not others (e.g., taking the balance on the last day of a period instead of summing across days).
This diagram shows the relationship between aggregation parameters and non-additive dimension configurations used for semi-additive metrics.
Sources: crates/dbt-metricflow/tests/metricflow_compat.rs:150-165, crates/dbt-parser/src/resolve/resolve_metrics.rs:504-515
Resolution and Manifest Generation
The resolve phase in resolver.rs is responsible for building the semantic layer by parsing YAML properties and models.
Integration with dbt Project
The resolver processes packages in waves, ensuring that dependencies are resolved before higher-level metrics are constructed. It specifically handles the transition from legacy metric specs to the modern semantic layer spec.
The sequence illustrates the order of resolution during the dbt parse/resolve phase.
Sources: crates/dbt-parser/src/resolver.rs:596-650, crates/dbt-parser/src/resolve/resolve_metrics.rs:90-120
Selective Jinja Rendering
Metrics use a restricted Jinja environment. While fields like description are rendered to support documentation (e.g., {{ doc('...') }}), fields like filter or expr are preserved as raw MetricFlow DSL to be evaluated later by the semantic compiler.
Sources: crates/dbt-parser/src/resolve/resolve_metrics.rs:40-60
Testing and Compatibility
The dbt-metricflow crate includes a comprehensive differential testing suite (metricflow_compat.rs) that validates the Rust compiler's output against MetricFlow's reference fixtures.
Dialect Support
The compiler is tested across multiple SQL dialects to ensure consistency. Validation checks for dialect-specific constructs, such as generate_series in DuckDB versus GENERATOR in Snowflake.
| Dialect | Key Characteristics |
|---|---|
| DuckDB | Uses generate_series for time spines; CAST(... AS DOUBLE). |
| Snowflake | Uses GENERATOR; supports IS NOT DISTINCT FROM for NULL-safe joins. |
| BigQuery | Uses FLOAT64 and CAST() instead of :: syntax. |
| Databricks | Uses CAST(); supports specific interval syntax. |
Sources: crates/dbt-metricflow/tests/metricflow_compat.rs:1250-1320, crates/dbt-metricflow/tests/data/convert_python_tests.py:65-75
Summary
The MetricFlow Semantic Layer provides dbt with powerful multi-dimensional modeling capabilities. By decoupling metric definitions from physical tables and using a specialized compiler to generate complex SQL (including multi-hop joins and time-spine windowing), it ensures that business logic remains portable and consistent across different data platforms. The architecture centers on a robust resolution phase that builds a semantic manifest used by the in-memory metric store and compiler for high-performance analytical queries.