Getting Started
Relevant source files
The following files were used as context for generating this wiki page:
dbt-common serves as the shared utility library containing the foundational code used by both dbt-core and various dbt adapter implementations. Its primary purpose is to provide a unified set of tools for logging, system interactions, Jinja rendering, and schema validation across the dbt ecosystem.
Sources: README.md:3-5, pyproject.toml:5-7, CONTRIBUTING.md:3-3
Development Environment Setup
To contribute to dbt-common, you must set up a local Python development environment. The project uses hatch for project management and environment orchestration.
Prerequisites
- Python: Version 3.10 or higher is required.
- Git: Required for source control and cloning the repository.
- Hatch: The primary tool for managing dependencies and running development scripts.
- Pip: Ensure the latest version is installed (
pip install --upgrade pip).
Sources: pyproject.toml:8-8, CONTRIBUTING.md:52-52, CONTRIBUTING.md:111-112
Required Tools
The project relies on several standard Python tools for code quality and maintenance:
| Tool | Purpose |
|---|---|
hatch | Project management and environment isolation |
flake8 | Code linting |
black | Code formatting (configured to 99 characters) |
mypy | Static type checking |
pre-commit | Running checks automatically before commits |
changie | Generating conflict-free changelog entries |
Sources: CONTRIBUTING.md:88-95, pyproject.toml:132-132
Installation Steps
- Fork and Clone: External contributors should fork the repository and clone it locally.
- Install Pre-commit: Run
pre-commit installto set up git hooks. - Verify Environment: Run
hatch env showto see available environments and commands.
Sources: CONTRIBUTING.md:68-74, CONTRIBUTING.md:131-131, CONTRIBUTING.md:140-140
Project Workflow
dbt-common follows a structured workflow for testing and code quality. The following diagram illustrates the typical development cycle for a contributor.
The diagram shows the transition from local development through testing and linting to the final pull request submission.
Sources: CONTRIBUTING.md:143-152, CONTRIBUTING.md:173-176, CONTRIBUTING.md:204-206
Core Modules Overview
Events and Logging
The Events module manages internal communication via a structured interface. It uses protobuf definitions (compiled via hatch run proto) to ensure machine-readable logs and event streams.
- Entry Point:
dbt_common.events.functions::fire_eventis the primary method for firing events. - Levels: Events are categorized into
DebugLevel,InfoLevel,WarnLevel, andErrorLevel. - Structure: Every event requires a unique
code, a log level, and amessage()method.
Sources: dbt_common/events/README.md:1-20, dbt_common/events/README.md:28-33
Jinja Rendering
The project provides specialized Jinja environments and parsers, such as MacroFuzzEnvironment and NativeSandboxEnvironment, to handle dbt-specific Jinja logic like macros and materializations.
The diagram represents the relationship between the custom Jinja environments used for template rendering.
Sources: dbt_common/clients/jinja.py:171-200, dbt_common/clients/jinja.py:228-229
System Operations
The dbt_common.clients.system module provides cross-platform utilities for file and directory management, including handling Windows long-path limitations. Key functions include:
find_matching(): Search for files using patterns.write_file()/load_file_contents(): Standardized I/O.run_cmd(): Executing subprocesses with event logging for STDOUT/STDERR.
Sources: dbt_common/clients/system.py:84-89, dbt_common/clients/system.py:202-202, dbt_common/clients/system.py:657-658
Hatch Command Reference
Commonly used commands for development tasks within the hatch environment:
| Command | Description |
|---|---|
hatch run test:unit | Executes all unit tests with coverage reporting |
hatch run lint:all | Runs black, flake8, and mypy checks |
hatch run proto | Regenerates Python classes from protobuf definitions |
hatch shell | Enters the default development shell |
changie new | Prompts for a new changelog entry |
Sources: CONTRIBUTING.md:143-152, pyproject.toml:117-117, CONTRIBUTING.md:174-176
Conclusion
dbt-common provides the essential scaffolding for the dbt ecosystem. By utilizing hatch and a strict event-driven architecture, it ensures consistency across different database adapters and the core execution engine. Developers should prioritize using the provided system and jinja clients to maintain cross-platform compatibility and proper event logging.