Environment Variables & Overrides
Relevant source files
The following files were used as context for generating this wiki page:
The flext-tap-ldif project utilizes a structured configuration system built on Pydantic and FlextSettingsBase. This system allows for the definition of runtime settings that can be controlled via application-specific environment variables or direct configuration passed during the initialization of the Singer Tap.
The primary purpose of this module is to provide validated, type-safe configuration for LDIF file extraction processes. This includes specifying source file paths, directory search patterns, encoding preferences, and parsing strictness. By using a centralized FlextTapLdifSettings class, the project ensures that all configuration overrides follow a predictable schema and prefixing convention.
Sources: src/flext_tap_ldif/settings.py:1-15, README.md:21-25
Configuration Architecture
The configuration system is anchored by the FlextTapLdifSettings class, which inherits from FlextSettingsBase. This architecture leverages Pydantic's SettingsConfigDict to automatically map environment variables to class attributes based on a specific prefix.
Environment Variable Prefixing
All environment variables intended to override the default settings must be prefixed with FLEXT_TAP_LDIF_. This prefixing ensures that the tap's configuration does not collide with other components in a Meltano or Singer environment.
The diagram shows how external environment variables and application-level configuration mappings are merged and validated before becoming an active FlextTapLdifSettings object.
Sources: src/flext_tap_ldif/settings.py:18-22, README.md:21-25
Supported Configuration Options
The following table details the specific variables available for override. These can be set via environment variables (e.g., FLEXT_TAP_LDIF_FILE_PATH) or through the Singer config.json.
| Setting | Type | Default | Description |
|---|---|---|---|
file_path | str | None | None | Explicit path to a single LDIF file. |
directory_path | str | None | None | Directory to search for LDIF files. |
file_pattern | str | *.ldif | Glob pattern for file discovery in a directory. |
encoding | str | utf-8 | Text encoding for reading files. |
strict_parsing | bool | True | If true, parsing errors will raise exceptions. |
max_file_size_mb | int | 100 | Maximum allowed size for a single file (>= 1MB). |
Sources: src/flext_tap_ldif/settings.py:24-41, src/flext_tap_ldif/_utilities/processor.py:27-40
Internal Logic & Validation
The settings module includes internal logic to normalize paths and enforce constraints on input values.
Path Normalization
The FlextTapLdifSettings class provides a helper method normalized_file_path() which converts the string-based file_path configuration into a pathlib.Path object. This ensures consistent cross-platform path handling within the Processor and DataProcessing utilities.
Data Flow for Discovery
When the Processor starts, it utilizes these settings to filter files based on size and pattern:
The sequence diagram illustrates how the max_file_size_mb setting is applied during the file discovery phase to prevent processing excessively large files.
Sources: src/flext_tap_ldif/settings.py:43-47, src/flext_tap_ldif/_utilities/processor.py:33-58
Parsing Behavior
The strict_parsing setting determines how the Processor.process_file method handles exceptions. If strict_parsing is enabled, any exception encountered during the yield of records (specifically those not categorized as SINGER_SAFE_EXCEPTIONS) will be re-raised, halting execution.
Sources: src/flext_tap_ldif/_utilities/processor.py:61-75
Implementation Details
The settings are defined using Annotated types and u.Field (an alias for Pydantic's Field) to provide metadata and validation constraints.
# From src/flext_tap_ldif/settings.py:38-41
max_file_size_mb: Annotated[
int,
u.Field(default=c.TapLdif.MAX_FILE_SIZE_MB, ge=1),
]
The use of ge=1 ensures that the maximum file size cannot be configured to a value less than 1 megabyte.
Sources: src/flext_tap_ldif/settings.py:38-41, src/flext_tap_ldif/_models/file_metadata.py:108-115
Summary
Environment variables and overrides in flext-tap-ldif are managed through a centralized, validated FlextTapLdifSettings class. By using the FLEXT_TAP_LDIF_ prefix, the system allows for flexible runtime configuration that remains isolated and safe. The integration with pydantic ensures that all overrides—whether for file paths, encoding, or performance limits like file size—are strictly typed and validated before they reach the core processing logic.