Event System Overview
Relevant source files
The following files were used as context for generating this wiki page:
The Event System in dbt-common is responsible for communicating internal dbt structures into a consumable interface for logging, streaming, and monitoring. It utilizes a structured approach where every event is defined by a unique class, often backed by Protobuf definitions to ensure a clearly defined interface regardless of the consumption method.
Sources: dbt_common/events/README.md:1-5, dbt_common/events/types.py:7-12
Core Architecture
The system is built around an EventManager that orchestrates the flow of events from the point of firing to various destinations like loggers and callbacks. Events are categorized by log levels and functional areas using a standardized coding scheme.
Event Definition and Levels
Every event must implement a unique code(), assign a log level via mixins (e.g., InfoLevel, WarnLevel), and provide a human-readable message().
| Level | Description | Mixin Class |
|---|---|---|
| DEBUG | Fine-grained informational events that are most useful to debug an application. | DebugLevel |
| INFO | Informational messages that highlight the progress of the application at coarse-grained level. | InfoLevel |
| WARN | Potentially harmful situations. | WarnLevel |
| ERROR | Error events that might still allow the application to continue running. | ErrorLevel |
Sources: dbt_common/events/base_types.py:165-195, dbt_common/events/README.md:21-27, dbt_common/events/types.py:15-26
System Components and Flow
The following diagram illustrates how an event moves from the application code through the management layer to final output destinations.
The fire_event function serves as the primary entry point, delegating the actual processing to the EventManager.
Sources: dbt_common/events/functions.py:133-145, dbt_common/events/event_manager.py:64-95
Event Management and Deferral
The EventManager handles the lifecycle of an event, including filtering, serialization, and support for deferred execution.
Deferred Events
Events can be queued instead of fired immediately using fire_or_defer_event. This is useful for grouping events (e.g., during parsing) to be processed or summarized later.
- Event Groups: Defined by
EventGroupType(e.g.,DEFAULT,PARSE). - Flush Mechanism:
fire_deferred_eventsprocesses all queued events for a specific group and raises a combinedEventCompilationErrorif any contained events were flagged as errors.
Sources: dbt_common/events/event_manager.py:108-142, dbt_common/events/base_types.py:38-44
Warning and Error Handling
The system supports sophisticated warning-as-error logic. The EventManager checks warn_error flags and warn_error_options to determine if a WarnLevel event should be elevated to an EventCompilationError or silenced entirely.
Sources: dbt_common/events/event_manager.py:70-80, dbt_common/events/functions.py:22-26
Logging and Formatting
The system supports multiple output formats and destinations managed through LoggerConfig.
Output Formats
Supported line formats are defined in the LineFormat Enum:
- PlainText: Standard human-readable console output.
- DebugText: Verbose output including timestamps, thread names, and invocation IDs.
- Json: Machine-readable structured logs, often used for external log aggregators.
Sources: dbt_common/events/logger.py:44-48, dbt_common/events/logger.py:133-176
Logger Configurations
Loggers are highly configurable, supporting filtering and "scrubbing" (redacting secrets from log messages).
Sources: dbt_common/events/logger.py:82-125
Data Serialization
Events are converted from internal Python classes to Protobuf messages (EventMsg) before being passed to loggers.
Metadata Enrichment
Every EventMsg is enriched with standard metadata including:
- Invocation ID: Unique identifier for the dbt run.
- Timestamp: UTC timestamp of the event.
- Process/Thread info: PID and thread name.
- Extra Metadata: Custom environment variables prefixed with
DBT_ENV_CUSTOM_ENV_.
Sources: dbt_common/events/base_types.py:118-141, dbt_common/events/functions.py:166-174
Structured Output (JSON)
For JSON logging, the system uses msg_to_dict and MessageToDict (from the Google Protobuf library) to ensure consistent field naming and presence.
Sources: dbt_common/events/functions.py:85-115
Summary
The dbt-common Event System provides a robust, type-safe framework for application telemetry. By separating event definitions (classes) from their transport (loggers/callbacks) and leveraging Protobuf for serialization, the system maintains high performance while offering the flexibility needed for both interactive CLI usage and automated environment monitoring.