Internal Schemas & Manifests
Relevant source files
The following files were used as context for generating this wiki page:
The internal schemas and manifests of the project define the core data structures used to represent a dbt project's state, transformation logic, and execution artifacts. These schemas provide a type-safe interface for parsing YAML configurations, managing the Directed Acyclic Graph (DAG) of project resources, and generating standard dbt artifacts such as manifest.json.
The system is designed to maintain parity with dbt-core while leveraging Rust's performance for the "Fusion" runtime. This involves resolving complex hierarchical configurations—merging project-level, package-level, and resource-level settings—into a final DbtManifest that represents the fully qualified state of the transformation environment.
Sources: crates/dbt-parser/src/resolver.rs:65-72, crates/dbt-schemas/src/schemas/manifest/manifest.rs:118-125
Manifest Architecture & Lifecycle
The manifest represents the single source of truth for a dbt project. It is constructed during the "Resolve" phase, where raw source files (models, macros, seeds, etc.) are parsed, validated, and linked together via dependencies.
The Resolver State
The ResolverState is an intermediate in-memory structure that holds all resolved nodes, macros, and operations before they are serialized into a formal manifest. It tracks:
- Nodes: Models, tests, snapshots, seeds, and analyses.
- Macros: User-defined and internal global project macros.
- Dependencies: Explicit parent/child relationships derived from
ref,source, andmetriccalls. - Metadata: Information about the adapter type, project name, and run start times.
The diagram shows the transition from raw project files to the final resolved state and the two primary output formats: the standard manifest.json and the Fusion-optimized Parquet index.
Sources: crates/dbt-parser/src/resolver.rs:77-110, crates/dbt-schemas/src/schemas/manifest/manifest.rs:128-140
Build Process
The build_manifest function transforms the ResolverState into a versioned DbtManifest (currently defaulting to V12). This process includes normalizing paths to be package-relative and generating deterministic parent/child maps.
| Component | Description |
|---|---|
ManifestMetadata | Contains schema versions, project IDs, and adapter-specific quoting policies. |
DbtNode | An enum wrapping various node types (Model, Test, Seed, etc.) into a uniform interface. |
Parent/Child Maps | Deterministic, sorted maps of unique IDs representing the DAG structure. |
Disabled Map | Tracks nodes that are explicitly disabled via configuration. |
Sources: crates/dbt-schemas/src/schemas/manifest/manifest.rs:68-80, crates/dbt-schemas/src/schemas/manifest/manifest.rs:128-150
Node Resolution & Configuration Merging
Nodes are resolved through a hierarchical configuration merge process. This ensures that specific overrides (e.g., in a schema.yml file) take precedence over general defaults (e.g., in dbt_project.yml).
Source Resolution Logic
Resolving a source requires merging information from multiple layers. For instance, source freshness rules are merged across the source and table levels rather than being simply overwritten.
The diagram illustrates the hierarchical order of configuration application for sources, noting that meta, tags, and freshness follow additive or field-level merging rather than total replacement.
Sources: crates/dbt-parser/src/resolve/resolve_sources.rs:89-105, crates/dbt-parser/src/resolve/resolve_sources.rs:136-150
Freshness Peer-Clearing
A critical implementation detail in source resolution is the mutual exclusion of loaded_at_field and loaded_at_query. If a table-level override sets one, it must implicitly clear any inherited value for the other from the source level to avoid validation errors.
Sources: crates/dbt-parser/src/resolve/resolve_sources.rs:430-450
Parquet Indexing & Merge-on-Write
The Fusion runtime uses a Parquet-based index for fast, read-only access to project metadata (used by systems like dbt-docs-server). Unlike the JSON manifest, which is usually rewritten entirely, the Parquet index supports efficient merging.
Write Modes
The IndexWriter supports several WriteMode variants to handle incremental updates without full re-computation:
| Mode | Behavior |
|---|---|
Overwrite | Default. New file replaces the old one. |
CarryForward | Merges new rows by primary key; carries forward specific column values (like compiled_code) if the new value is NULL. |
MergePrune | Keeps old rows not present in the new batch, provided their IDs are still valid in the project. |
ReplaceColumnLineage | Replaces lineage for recomputed targets while preserving last-known lineage for others. |
Sources: crates/dbt-index-core/src/parquet.rs:114-140
Data Model for Index Tables
Parquet tables are defined using a define_row! macro that generates both Rust serialization structs and Arrow schemas.
define_row! {
/// dbt.node_columns row.
pub struct NodeColumnRow<'a> => schema_node_columns {
[utf8!] pub unique_id: &'a str,
[utf8!] pub column_name: &'a str,
[int] pub column_index: Option<i64>,
[utf8] pub declared_type: Option<&'a str>,
[utf8] pub inferred_type: Option<&'a str>,
[utf8] pub catalog_type: Option<&'a str>,
[utf8] pub data_type: Option<&'a str>,
// ... additional fields
[timestamp !] pub ingested_at: &'a str,
}
}
Sources: crates/dbt-index-core/src/parquet.rs:608-620, crates/dbt-index-core/src/parquet.rs:770-785
Artifact Comparison & State
The internal schemas facilitate state-based features like state:modified.
Checksum Recalculation
Checksums are used to detect changes in models and snapshots. For SQL-based nodes, the project applies whitespace and case normalization to the raw code before hashing. This ensures that formatting changes do not trigger a "modified" status in the DAG.
Sources: crates/dbt-schemas/src/schemas/manifest/manifest.rs:990-1010
Path Normalization
Manifest paths are normalized to ensure consistent behavior across different environments. original_file_path remains project-relative, while path is serialized relative to the configured resource root (e.g., the directory listed in model-paths).
Sources: crates/dbt-schemas/src/schemas/manifest/manifest.rs:373-385
Conclusion
The project's internal schemas and manifests form a robust backbone for transformation management. By combining a strict Rust-based resolution engine with standard dbt-core artifact compatibility and a high-performance Parquet indexing system, the project enables fast DAG resolution and metadata access while ensuring that complex configuration inheritance rules are correctly enforced. This dual-path approach supports both traditional dbt workflows and modern, low-latency metadata requirements.