Stream Processing Lifecycle
Relevant source files
The following files were used as context for generating this wiki page:
The Stream Processing Lifecycle in flext-target-ldif defines the end-to-end journey of Singer data records as they are ingested, transformed, and persisted into the LDAP Data Interchange Format (LDIF). This lifecycle ensures that data extracted from various sources is correctly mapped to LDAP structures, validated against business rules, and written to the filesystem with proper formatting such as line folding and Base64 encoding for non-ASCII characters.
The system utilizes a structured approach involving target initialization, sink creation per stream, and a buffered writing process that culminates in the generation of compliant LDIF files.
Sources: src/flext_target_ldif/target.py:1-20, src/flext_target_ldif/models.py:10-30
Target Initialization and Configuration
The lifecycle begins with the instantiation of FlextTargetLdif. During initialization, the target merges user-provided settings with internal defaults, including naming patterns for files and default Distinguished Name (DN) templates. A critical step in this phase is the creation of the output directory where LDIF files will reside.
The configuration determines global behavior for the entire processing run, such as whether to force Base64 encoding or what maximum line length to enforce for LDIF compliance.
Sources: src/flext_target_ldif/target.py:28-48, src/flext_target_ldif/models.py:45-72
Stream and Sink Management
For every unique Singer stream, the target dynamically allocates a Sink. The sink acts as the state manager for a specific stream's data flow. It holds references to the stream schema, the specific FlextTargetLdifWriter instance, and the target file path.
Sink Creation Logic
When a stream is first encountered, the get_sink method is called. If a sink does not exist, it is created using FlextTargetLdifModels.TargetLdif.Sink.
Sources: src/flext_target_ldif/target.py:65-74, src/flext_target_ldif/_utilities/service_runtime.py:59-86
| Component | Responsibility |
|---|---|
FlextTargetLdif | Manages the global lifecycle and sink registry. |
TargetLdif.Sink | Handles batch and record processing for a specific stream. |
ServiceRuntime | Provides normalization and adaptation between service and domain layers. |
Record Transformation and Validation
Before a record is written to the buffer, it undergoes transformation to ensure LDAP compatibility. This includes mapping source fields to LDAP attributes and generating a mandatory Distinguished Name (DN).
Transformation Flow
The RecordTransformer utility provides specialized logic for different data types:
- DN Generation: Uses a template (e.g.,
uid={uid},ou=users,dc=example,dc=com) to build the entry's identity. - Attribute Mapping: Translates record keys to LDAP-specific names (e.g.,
emailtomail). - Type Normalization: Converts booleans to
TRUE/FALSEand formats timestamps.
Data Persistence and File Writing
The FlextTargetLdifWriter manages the physical file I/O. To optimize performance and ensure file consistency (like including a version header and timestamp), the writer buffers records in memory and performs a bulk write during the close() operation.
LDIF Formatting Rules
The writer enforces several RFC 2849 requirements:
- Base64 Encoding: Automatically applied to values starting with unsafe characters (space, colon, less-than) or containing non-ASCII/newlines.
- Line Folding: Wraps lines exceeding the configured
line_length(default 78) by adding a leading space on subsequent lines. - Version Header: Prefixes every file with
version: 1.
Sources: src/flext_target_ldif/writer.py:157-220, src/flext_target_ldif/utilities.py:126-150
Lifecycle Termination
The lifecycle concludes when the Singer process signals the end of a stream. The Sink.clean_up() method triggers the LDIFWriter.close() process.
During closure, the writer iterates through its internal record buffer, performs final entry conversion, and writes the complete LDIF block to the filesystem.
Sources: src/flext_target_ldif/models.py:214-228, src/flext_target_ldif/writer.py:84-101
Summary
The Stream Processing Lifecycle in flext-target-ldif provides a robust pipeline for converting stream-based JSON data into standardized LDIF files. By decoupling configuration, transformation, and I/O logic across the Target, Sink, and Writer components, the system maintains Singer protocol compliance while strictly adhering to LDAP format specifications.
Sources: src/flext_target_ldif/target.py, src/flext_target_ldif/models.py, src/flext_target_ldif/writer.py