System & Connection Clients
Relevant source files
The following files were used as context for generating this wiki page:
The System and Connection modules within dbt-common provide a standardized interface for interacting with the host operating system and managing external calls. These utilities ensure consistent behavior across different platforms (macOS, Linux, and Windows) while handling file I/O, command execution, and package extraction.
The system client acts as a wrapper around Python's standard libraries like os, shutil, and subprocess, adding robust error handling and logging via dbt's event system. It is designed to be used by both dbt-core and various adapter implementations to maintain a unified approach to environment interaction. Sources: README.md, CONTRIBUTING.md:1-5, dbt_common/clients/system.py
System Client Operations
The System Client manages fundamental file system operations, including file creation, directory management, and file searching. It incorporates security measures to prevent path traversal and handles platform-specific differences in command execution.
File and Directory Management
The client provides functions to safely create files and directories, ensuring that existing files are not overwritten unless explicitly requested.
| Function | Purpose | Key Parameters |
|---|---|---|
make_file | Creates a file with specific contents. | path, contents, overwrite |
make_directory | Recursively creates directories. | path |
rename | Moves or renames a file/directory. | from_path, to_path |
find_matching | Searches for files based on patterns and ignore specs. | root_path, relative_paths, pattern, ignore_spec |
The rename function includes logic to handle cases where the source and destination paths are identical, preventing accidental deletion. Sources: tests/unit/test_system_client.py:42-81, CHANGELOG.md:85-87
Secure Package Extraction
The system client provides a untar_package utility for extracting compressed archives. This utility includes critical security checks to prevent "Zip Slip" style path traversal attacks.
The extraction process uses os.path.commonpath() instead of the deprecated os.path.commonprefix() to ensure that members cannot be extracted to sibling directories that share a naming prefix with the destination. Sources: CHANGELOG.md:43-45, tests/unit/test_system_client.py:195-236
Command Execution
The client facilitates running shell commands through the run_cmd function. This function abstracts subprocess.Popen and provides detailed feedback through standard output, standard error, and return codes.
Execution Flow and Error Handling
When a command is executed, the client validates both the executable path and the working directory.
- Validation: Checks if the working directory exists and is accessible.
- Execution: Invokes the command and captures output streams.
- Logging: Fires events for command execution, STDOUT, STDERR, and return codes.
- Exceptions: Raises
ExecutableErrorif the command is missing/not executable, orWorkingDirectoryErrorif the CWD is invalid.
Connection and External Call Utilities
While primarily focused on system-level tasks, the utilities support external call management, particularly regarding retries and exception handling for network-based operations.
Retry Logic
The system supports retrying external calls, which is signaled via specific debug-level events. This is used to manage transient failures during operations like package downloading or adapter-specific network requests.
| Event Class | Code | Description |
|---|---|---|
RetryExternalCall | M020 | Logs the current attempt and maximum allowed attempts. |
RecordRetryException | M021 | Logs the specific exception that triggered the retry. |
Sources: dbt_common/events/types.py:80-92, CHANGELOG.md:112-114
Exception Hierarchy
The system and connection operations utilize a structured exception hierarchy to provide granular error reporting.
Key System Exceptions
- DbtRuntimeError: The base class for most operational failures, supporting stack traces and node identification.
- CommandError: Specifically for shell command failures, automatically scrubbing secrets from the command text.
- ExecutableError: Raised when a requested command cannot be found or executed.
- WorkingDirectoryError: Raised when the specified
cwdfor a command is missing or lacks permissions.
Sources: dbt_common/exceptions/base.py:59-158, tests/unit/test_system_client.py:105-135
Secret Scrubbing
To prevent the accidental leakage of sensitive information (like API keys or passwords) in logs or error messages, the system client uses a scrub_secrets utility. This utility scans messages for any environment variables prefixed with DBT_ENV_SECRET_ and replaces them with *****. Sources: dbt_common/exceptions/base.py:10-22
Summary
The System & Connection Clients in dbt-common serve as a hardened abstraction layer for OS-level interactions. By centralizing logic for path validation, secure extraction, and command execution, the module provides a consistent and secure foundation for dbt's core engine and its various database adapters. Sources: README.md, CHANGELOG.md