Event Formatting & Logging

The Event Formatting & Logging system in dbt-common provides a structured framework for communicating internal state and occurrences into consumable interfaces. It is designed to bridge the gap between human-readable console output and machine-reliable event streams by utilizing a unified event model based on Protobuf definitions.

The system handles the entire lifecycle of an event, from instantiation and metadata enrichment to filtering, scrubbing of sensitive information, and final serialization into various formats like Plain Text or JSON. This ensures that all events—ranging from project parsing to database execution—are logged consistently across files, stdout, and external callbacks.
Sources: dbt_common/events/README.md:1-12, dbt_common/events/types.py:7-15

Event Lifecycle and Management

The logging process is orchestrated by the EventManager, which serves as the central hub for dispatching events to registered loggers and callbacks. The entry point for this system is the fire_event function.

Event Firing Flow

When an event is fired, the system performs the following steps:

1. Message Creation: Converts a BaseEvent into a structured EventMsg containing metadata (timestamp, PID, thread name, invocation ID).
2. Validation & Handling: Checks if the event should be treated as an error based on WarnErrorOptions.
3. Distribution: Iterates through registered loggers and executes their filters. If an event passes the filter, it is written to the output stream.
4. Callbacks: Executes any registered TCallback functions.

The diagram shows the sequence of operations from the initial event call to the final output.
Sources: dbt_common/events/event_manager.py:61-94, dbt_common/events/base_types.py:126-150

Core Components

Logger Configurations

The behavior of each logger is defined by a LoggerConfig object. This configuration determines the output destination, formatting style, and filtering logic.

Sources: dbt_common/events/logger.py:64-77, dbt_common/events/functions.py:46-60

Formatting Logic

Logging supports multiple formats tailored for different consumption needs:

• PlainText: Simplified human-readable format, often omitting timestamps for specific "Print" events to support clean CLI output.
• DebugText: Enhanced format including timestamps, log levels, and thread names, used primarily for file logging.
• JSON: Full machine-readable serialization of the event Protobuf, including both info (metadata) and data (event-specific fields).

# PlainText formatting example
ts = datetime.now(timezone.utc).strftime("%H:%M:%S")
log_line = f"{ts}  {scrubbed_msg}"

# DebugText formatting example
log_line = f"{ts} [{level:<5}] [{thread_name}] {scrubbed_msg}"

Sources: dbt_common/events/logger.py:129-165, dbt_common/events/functions.py:86-107

Specialized Event Types

The system categorizes events through a prefix-based coding scheme and severity-based class hierarchy.

Severity Mixins

Events inherit from specific mixins to define their default log level:

• DebugLevel: Used for internal diagnostic information.
• InfoLevel: Standard operational messages.
• WarnLevel: Potential issues that do not halt execution.
• ErrorLevel: Critical failures.

Event Code Categories

Event codes translate to the natural order of task execution:

Sources: dbt_common/events/types.py:18-35, dbt_common/events/base_types.py:192-215

Security and Scrubbing

A critical phase of the formatting process is "scrubbing." Before any line is written to a logger, it is passed through a Scrubber. The env_scrubber utilizes scrub_secrets and env_secrets to ensure that any sensitive information (e.g., credentials) present in the event message or data is redacted.
Sources: dbt_common/events/functions.py:72-74, dbt_common/events/logger.py:44-51

Event Deferral

The EventManager supports a deferral mechanism, allowing events to be queued and processed later. This is particularly useful during project parsing where errors might need to be aggregated before being raised to the user.

The Event Formatting & Logging system provides a robust, extensible foundation for dbt's observability, ensuring that whether an operator is looking at a terminal or an automated system is parsing JSON logs, the information is accurate, secure, and well-structured.