Record & Replay Mechanism
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Record & Replay mechanism in dbt-common provides a framework for capturing dbt's interactions with external systems during a command invocation and re-running those commands later using the recorded data. This allows for deterministic testing and debugging by "replaying" previously captured results instead of executing real external requests.
Sources: dbt_common/record.py:1-6, CHANGELOG.md:195-197
The system operates in three primary modes: RECORD, where interactions are saved to a file; REPLAY, where saved data is returned in place of real execution; and DIFF, which performs recording and compares it against a previous run to identify changes.
Sources: dbt_common/record.py:99-102
Core Components
The mechanism relies on several key classes to manage data structures and execution flow.
Record Data Structure
A Record represents a single interaction, consisting of parameters (params), a result (result), and a sequence number (seq) to maintain order.
The Recorder Class
The Recorder is the central coordinator. It handles adding new records, popping matching records during replay, and persisting data to disk. It supports both in-memory storage and streamed writing to a file (defaulting to recording.json).
Sources: dbt_common/record.py:105-147
Implementation Mechanisms
Decorators
The system uses decorators to wrap functions that should be recorded.
| Decorator | Description |
|---|---|
@record_function | Manually wraps a function using a specific Record class. Supports tracking thread IDs. |
@auto_record_function | Automatically generates Record, Params, and Result boilerplate classes for simpler functions. |
Sources: dbt_common/record.py:321-356
Class Support
The @supports_replay class decorator ensures that even if a subclass overrides a recorded method, the recording logic is preserved via __init_subclass__ manipulation.
Sources: dbt_common/record.py:539-586
Data Flow and Logic
Recording Flow
When a decorated function is called in RECORD mode, the record_replay_wrapper captures the input arguments, executes the function, captures the return value, and sends a Record object to the Recorder.
Replay Flow
In REPLAY mode, the wrapper avoids calling the original function. It instead creates a parameter object and asks the Recorder for a matching record.
Sources: dbt_common/record.py:214-228, dbt_common/record.py:441-443
Configuration and Environment Variables
The mechanism is configured primarily through environment variables, supporting both DBT_RECORDER_ and DBT_ENGINE_RECORDER_ prefixes.
| Variable | Description | Default |
|---|---|---|
DBT_RECORDER_MODE | Sets the mode: RECORD, REPLAY, or DIFF. | None |
DBT_RECORDER_FILE_PATH | Path to the JSON recording file. | recording.json |
DBT_RECORDER_TYPES | Comma-separated list of record types or groups to include. | All |
DBT_RECORDER_ROW_LIMIT | Limit on the number of rows recorded for specific data types. | None |
Sources: dbt_common/record.py:255-309, CHANGELOG.md:46-47
Diagnostics and Events
The system integrates with the dbt-common event system to report issues and external calls.
- M021 (RecordRetryException): Debug level event reporting external call exceptions.
- Z053 (RecordReplayIssue): General info level event for reporting runtime issues with the record/replay system.
Sources: dbt_common/events/types.py:72-76, dbt_common/events/types.py:155-161
Conclusion
The Record & Replay mechanism provides a robust way to isolate dbt executions from external environments. By utilizing Python decorators and serialization strategies (via mashumaro), it enables deterministic re-runs of complex tasks, which is critical for regression testing and CI/CD pipelines in dbt environments.