Home

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• CONTRIBUTING.md
• dbt_common/clients/jinja.py
• dbt_common/contracts/config/base.py
• dbt_common/dataclass_schema.py
• dbt_common/events/types.py
• dbt_common/semver.py
• dbt_common/exceptions/base.py

dbt-common is a shared library containing common utilities, contracts, and core logic used by dbt-core and various dbt adapter implementations. Its primary purpose is to provide a unified foundation for event handling, Jinja2 template rendering, configuration management, and semantic versioning across the dbt ecosystem.

The repository is structured to support complex dbt operations, including specialized Jinja extensions for materializations and documentation, a robust schema validation system based on mashumaro, and a comprehensive event-driven logging architecture.

Sources: README.md:3-5, CONTRIBUTING.md:3-4, pyproject.toml:6-8

Core Modules and Architecture

The library is organized into several key functional areas that facilitate the dbt execution lifecycle.

Jinja2 Client and Extensions

The Jinja2 implementation in dbt-common extends standard Jinja2 behavior to support dbt-specific constructs. This includes custom environments like MacroFuzzEnvironment and NativeSandboxEnvironment, which allow for specialized template rendering and macro parsing.

Key components include:

• Custom Parsers: MacroFuzzParser enables typed parameters in dbt macros.
• Extensions: Specialized extensions for materialization, docs, and test blocks.
• Caching: A TemplateCache mechanism to improve performance during large-scale compilation.

The diagram shows the relationship between the custom Jinja environment, the specialized parser, and the resulting dbt-prefixed macro names.
Sources: dbt_common/clients/jinja.py:91-110, dbt_common/clients/jinja.py:382-450, dbt_common/utils/jinja.py:10-37

Dataclass Schema and Validation

dbt-common utilizes mashumaro and jsonschema to provide strict type validation and serialization for dbt objects. The dbtClassMixin serves as the base for most contracts, enabling JSON schema generation and validation against data dictionaries.

Class	Description
dbtClassMixin	Base mixin for validation and serialization using Mashumaro.
DateTimeSerialization	Strategy for ISO8601 datetime serialization/deserialization.
StrEnum	Enums that behave as strings for serialization compatibility.
ValidatedStringMixin	String types validated against a specific regex.

Sources: dbt_common/dataclass_schema.py:64-90, dbt_common/dataclass_schema.py:157-175

Event System

The event system is built on top of protobuf definitions (via dbt-protos) to ensure consistent communication of internal dbt structures. Events are classified by levels (Debug, Info, Warn, Error) and assigned unique codes based on the project lifecycle.

Prefix	Description	Example Code
D	Deprecations	D000 (BehaviorChangeEvent)
M	Deps generation	M020 (RetryExternalCall)
Z	Miscellaneous	Z017 (Formatting)

Sources: dbt_common/events/types.py:15-30, dbt_common/events/README.md:1-5

Configuration Management

The BaseConfig class provides the logic for merging and accessing configuration values. It supports complex merge behaviors, allowing dbt to combine project-level, adapter-level, and node-level configurations.

Merge Behaviors

Configurations are merged according to specific rules defined in MergeBehavior:

1. Clobber: The new value completely replaces the old value.
2. Append: Lists are combined (e.g., tags, hooks).
3. Update: Dictionaries are merged (e.g., meta, quoting).
4. DictKeyAppend: Dictionaries containing lists are merged by extending the lists (e.g., grants).

Conceptual flow of how different configuration keys are handled during a merge operation.
Sources: dbt_common/contracts/config/base.py:102-115, dbt_common/contracts/config/base.py:199-230

Semantic Versioning

The library includes a custom implementation of semantic versioning (semver.py) tailored for dbt's requirement to handle version ranges and compatibility checks for packages and core versions.

Key functionalities include:

• VersionSpecifier: Parses and compares individual versions including pre-releases.
• VersionRange: Manages bounds (> , <=) and reduces multiple version requirements into a single compatible range.
• Resolution: Logic to find the highest installable version within a given range from a set of available versions.

Sources: dbt_common/semver.py:46-75, dbt_common/semver.py:321-340

Exception Handling

dbt-common defines a hierarchy of exceptions to provide clear error reporting. A critical feature of the exception system is the Secret Scrubbing mechanism, which prevents sensitive environment variables (prefixed with DBT_ENV_SECRET_) from appearing in logs or error messages.

def scrub_secrets(msg: Any, secrets: List[str]) -> Any:
    scrubbed = str(msg)
    for secret in secrets:
        scrubbed = scrubbed.replace(secret, "*****")
    return msg if str(msg) == scrubbed else scrubbed

Sources: dbt_common/exceptions/base.py:10-22, dbt_common/exceptions/base.py:64-70

Development and Contribution

The project uses hatch for environment management and testing. Contributors must sign a Contributor License Agreement (CLA) before pull requests can be merged.

• Testing: Unit tests are executed via hatch run test:unit.
• Linting: Strict linting is enforced using black, flake8, and mypy.
• Changelog: Managed via changie to prevent merge conflicts in the CHANGELOG.md file.

Sources: CONTRIBUTING.md:47-60, CONTRIBUTING.md:120-135