Event Manager & Routing
Relevant source files
The following files were used as context for generating this wiki page:
The Event Manager & Routing system serves as the central hub for communicating internal dbt structures into a consumable interface. It is responsible for intercepting events fired within the application and routing them to various destinations, including human-readable logs, machine-reliable event streams, and registered callbacks. The system is built upon protobuf definitions to ensure a clearly defined interface for all events.
Sources: dbt_common/events/README.md:1-5, dbt_common/events/event_manager.py:30-45
Core Architecture
The architecture centers around the EventManager class, which manages a collection of loggers and callbacks. When an event is "fired," the manager converts the base event into a structured EventMsg, evaluates filtering logic, and distributes the message to all active output channels.
Event Management Flow
The following diagram illustrates the lifecycle of an event from the initial call to fire_event through its final routing to loggers and callbacks.
This flow ensures that events are validated against warning/error configurations before reaching any output stream.
Sources: dbt_common/events/event_manager.py:65-95, dbt_common/events/functions.py:126-140
Key Components
The system relies on several core classes and structures to manage event data and routing behavior.
EventManager Class
The EventManager is the primary orchestrator. It maintains state regarding loggers, callbacks, and global warning/error settings.
| Component | Description |
|---|---|
loggers | A list of _Logger instances (Text or JSON) that format and write event lines. |
callbacks | A list of TCallback functions executed for every event. |
warn_error | Boolean flag indicating if warnings should be treated as errors. |
allow_deferral | Boolean flag allowing events to be queued rather than fired immediately. |
Sources: dbt_common/events/event_manager.py:23-30, dbt_common/events/event_manager.py:108-112
Event Deferral and Grouping
Events can be deferred using the fire_or_defer_event method. This is particularly useful during processes like parsing where multiple issues might need to be aggregated before raising a single exception.
Event Routing Logic
Routing is handled by two main mechanisms: Loggers and Callbacks.
Logger Configurations
Loggers are initialized via LoggerConfig and can target standard output or rotating files. The system supports three primary line formats defined in LineFormat: PlainText, DebugText, and Json.
# From dbt_common/events/logger.py
class LineFormat(Enum):
PlainText = 1
DebugText = 2
Json = 3
Sources: dbt_common/events/logger.py:48-52, dbt_common/events/event_manager.py:97-101
Filtering and Scrubbing
Before an event is logged, it passes through a Filter and a Scrubber. Filters determine if an event meets specific criteria (e.g., log level), while Scrubbers ensure that sensitive information (secrets) is removed from the output string.
Sources: dbt_common/events/logger.py:27-45, dbt_common/events/functions.py:56-58
Warning and Error Handling
The Event Manager integrates with WarnErrorOptions to dynamically promote warnings to errors or silence them entirely based on event codes or names.
| Function | Logic |
|---|---|
warn_error | If True, any event with a warn level triggers an EventCompilationError. |
silenced(e) | If an event is included in the silence list, the manager returns early without routing. |
force_warn_or_error_handling | Bypasses standard level checks to force evaluation against warn/error rules. |
Sources: dbt_common/events/event_manager.py:70-80, dbt_common/events/functions.py:102-104
Implementation Details
Serialization
Events are based on protobuf messages. The system provides utilities like msg_to_json and msg_to_dict to facilitate machine-readable exports. For debugging, the environment variable DBT_TEST_BINARY_SERIALIZATION can be used to verify that all fired events are binary-serializable.
Sources: dbt_common/events/functions.py:73-100, dbt_common/events/event_manager.py:82-90
Global Access
The event_manager_client.py provides a context-managed global instance of the EventManager. Dependents typically access the system via get_event_manager().
Sources: dbt_common/events/event_manager_client.py:1-20, dbt_common/events/functions.py:141-145
Conclusion
The Event Manager & Routing module provides a robust, extensible framework for internal communication within dbt. By separating event definitions from their final output format and destination, it allows for simultaneous human logging, structured data streaming, and complex error aggregation through its deferral mechanism.