Invocation & Context Management

Relevant source files

The following files were used as context for generating this wiki page:

• dbt_common/invocation.py
• dbt_common/events/functions.py
• dbt_common/events/types.py
• dbt_common/dataclass_schema.py
• CHANGELOG.md
• dbt_common/exceptions/base.py

The Invocation & Context Management system in dbt-common provides a standardized framework for tracking and managing the execution state of dbt tasks. It ensures that every execution (invocation) has a unique identity and consistent metadata, such as start times and environment variables, which are accessible across different modules like event logging and error handling.

This system is critical for multi-threaded execution environments and testing scenarios where multiple distinct dbt "runs" might overlap or require isolation. It facilitates the aggregation of telemetry, debugging information, and the preservation of global state without relying on unsafe global variables.
Sources: dbt_common/events/functions.py:27-40, CHANGELOG.md:167-175, CHANGELOG.md:237-238

Invocation Lifecycle and Identity

Every time dbt is executed, an invocation context is established. This context is identified by a unique invocation_id, which is used to correlate logs, events, and errors to a specific execution instance.

Execution Metadata

The system tracks several key timestamps and identifiers for each invocation:

• Invocation ID: A unique string generated for the specific run.
• Started At: The precise time the invocation commenced, often stored in the invocation_started_at field.
• Environment Context: Metadata derived from environment variables, particularly those prefixed with DBT_ENV_CUSTOM_ENV_.

The diagram shows the sequence of steps taken to initialize a new invocation context.
Sources: dbt_common/events/functions.py:27-40, dbt_common/events/functions.py:155-164, CHANGELOG.md:167-175

Event System Integration

The invocation context is heavily utilized by the EventManager to enrich logs with execution-specific data. This ensures that every event fired can be traced back to its parent invocation.

Logger Configuration

When configuring loggers (such as for stdout), the invocation_id is explicitly passed into the LoggerConfig. This allows for filtering and grouping of log lines by run.

Component	Description	Source
get_stdout_config	Generates configuration for stdout logging including invocation_id.	dbt_common/events/functions.py:32-48
invocation_id	Unique ID retrieved via get_invocation_id() during logger setup.	dbt_common/invocation.py
LoggerConfig	Dataclass that holds the invocation identity for a specific log stream.	dbt_common/events/functions.py:40

Sources: dbt_common/events/functions.py:32-48, dbt_common/events/types.py:127-133

Environment Variable Management

Invocation management includes handling custom environment variables that influence dbt's behavior. The system provides utilities to extract variables prefixed with DBT_ENV_CUSTOM_ENV_ into a structured dictionary.

Custom Metadata Handling

The get_metadata_vars function performs a one-time extraction of these environment variables, caching them in a global metadata_vars dictionary for the duration of the invocation.

_METADATA_ENV_PREFIX = "DBT_ENV_CUSTOM_ENV_"

def get_metadata_vars() -> Dict[str, str]:
    global metadata_vars
    if metadata_vars is None:
        metadata_vars = {
            k[len(_METADATA_ENV_PREFIX) :]: v
            for k, v in os.environ.items()
            if k.startswith(_METADATA_ENV_PREFIX)
        }
    return metadata_vars

Sources: dbt_common/events/functions.py:155-164

Error Context and Secret Scrubbing

During an invocation, errors may occur that contain sensitive information (secrets). The context management system integrates with the exception handling layer to ensure that secrets identified in environment variables (prefixed with DBT_ENV_SECRET_) are scrubbed from error messages before they are logged or displayed.

Secret Scrubbing Flow

1. Identify Secrets: The system scans os.environ for keys starting with SECRET_ENV_PREFIX.
2. Exception Interception: When a DbtRuntimeError or DbtInternalError is raised, the message is passed through scrub_secrets.
3. Redaction: Any substring matching a known secret is replaced with *****.

The diagram illustrates the flow of a message through the scrubbing utility to prevent secret leakage.
Sources: dbt_common/exceptions/base.py:10-21, dbt_common/exceptions/base.py:44-46, dbt_common/exceptions/base.py:74-76

Summary

Invocation & Context Management provides the glue that binds execution identity to logging, environment configuration, and error reporting. By centralizing the invocation_id and start-time tracking, dbt-common allows for sophisticated monitoring and debugging of dbt runs across both local and distributed execution environments. The system further protects security through integrated secret scrubbing and provides extensibility via custom environment variable mapping.
Sources: CHANGELOG.md:167-175, dbt_common/invocation.py, dbt_common/events/functions.py:32-48