Event Types & Protobuf Generation
Relevant source files
The following files were used as context for generating this wiki page:
The event system in dbt-common is responsible for communicating internal dbt structures into a consumable, structured interface. Every event in dbt is defined by a Protocol Buffer (protobuf) message definition, ensuring a clearly defined interface regardless of whether the consumer uses protobuf directly or other formats like JSON or plain text.
Sources: dbt_common/events/README.md:2-5, dbt_common/events/types.py:10-14
This module facilitates exhaustive logging and monitoring of dbt's execution by mapping Python-based event classes to generated protobuf types. These types support both human-readable logs and machine-reliable event streams, utilizing a standardized schema for metadata such as timestamps, invocation IDs, and process information.
Sources: dbt_common/events/base_types.py:101-118, dbt_common/events/README.md:8-10
Event Architecture & Base Types
The event system is built upon a hierarchy of base types that enforce the required structure for all concrete events. Every event is an instance of a class that inherits from BaseEvent and a specific level mixin (e.g., InfoLevel, WarnLevel).
Class Hierarchy
Events are categorized by their severity levels, which are mapped to standard logging levels.
Event Components
An event message (EventMsg) consists of two primary components:
- Info (
EventInfo): Standardized metadata common to all events (e.g.,ts,level,invocation_id,pid). - Data: The event-specific payload defined in the protobuf message.
| Field | Description | Source |
|---|---|---|
ts | UTC timestamp of the event. | base_types.py:106 |
level | The string value of the EventLevel. | base_types.py:105 |
invocation_id | Unique ID for the dbt run. | base_types.py:108 |
pid | Process ID firing the event. | base_types.py:111 |
code | Unique event code (e.g., "Z001"). | base_types.py:113 |
extra | Global metadata variables from environment. | base_types.py:109 |
Protobuf Generation & Compilation
Events are based on protobuf definitions compiled into Python classes using the Betterproto or standard protoc compilers.
Compilation Process
While the primary event protos are maintained in a central repository, they are compiled into types_pb2.py within the local environment to provide the underlying message structures used by BaseEvent.
Data Serialization Flow
When an event is fired, the Python object is transformed into a structured message format.
Event Categorization & Naming
Events follow a strict naming and coding convention. The unique event code consists of a prefix indicating the functional area of dbt where the event originated.
| Prefix | Description |
|---|---|
| A | Pre-project loading |
| D | Deprecations |
| E | DB adapter |
| I | Project parsing |
| M | Deps generation |
| P | Artifacts |
| Q | Node execution |
| W | Node testing |
| Z | Miscellaneous |
| T | Test only |
Sources: dbt_common/events/types.py:17-30
Example Event Definition
A concrete event requires a unique code, a message() for logging, and a level mixin.
class RetryExternalCall(DebugLevel):
def code(self) -> str:
return "M020"
def message(self) -> str:
return f"Retrying external call. Attempt: {self.attempt} Max attempts: {self.max}"
Sources: dbt_common/events/types.py:55-60, dbt_common/events/README.md:22-24
Implementation Details
Serialization to JSON
Machine-readable output is generated via msg_to_json or the _JsonLogger. The process uses the protobuf MessageToDict utility to ensure all fields, including those with default values, are represented.
Sources: dbt_common/events/functions.py:75-80, dbt_common/events/logger.py:183-189
Handling Secrets
The env_scrubber function is used during the logging phase to ensure that sensitive information found in environment variables is removed from the event strings before they are written to disk or stdout.
Sources: dbt_common/events/functions.py:61-63, dbt_common/events/logger.py:35-37
Manual Protobuf Testing
Unit tests verify that events can be serialized to binary and reconstituted, ensuring the Python classes correctly map to the types_pb2.py structures.
def test_events() -> None:
event = RetryExternalCall(attempt=3, max=5)
msg = msg_from_base_event(event)
serialized = msg.SerializeToString()
# Verification of binary serialization and info extraction
Sources: tests/unit/test_proto_events.py:27-37
Summary
The "Event Types & Protobuf Generation" system provides a robust, typed infrastructure for dbt's internal communication. By leveraging protobuf definitions as the source of truth, dbt-common ensures consistent data structures across human-readable logs and machine-consumable JSON streams, while enforcing metadata standards and functional categorization through a structured Python class hierarchy.