Structlog Context-Aware Logging
Relevant source files
The following files were used as context for generating this wiki page:
Structlog Context-Aware Logging in the FLEXT platform provides a structured approach to observability, ensuring that log entries carry consistent metadata and correlation context across distributed data integration workloads. It is integrated into the core architecture to support the platform's mission of building typed, validated, and reusable integration primitives.
The system utilizes structured logging to track execution metrics, correlation IDs, and operational context, which is essential for enterprise-grade observability in complex workflows such as LDAP migrations or Oracle database integrations. This logging strategy aligns with the "Continuous Green" and "Truth with Evidence" mandates of the FLEXT governance model.
Sources: AGENTS.md:215-220, README.md:1-10, examples/advanced_processing_example.py:1-12
Observability Architecture
The logging architecture is built upon the FLEXT Core foundations, utilizing a facade-based approach where common operational aliases (e.g., c, m, t, p, u) provide a consistent interface for utilities, models, and types. This ensures that context remains attached to the execution flow even during parallel processing.
Context and Correlation Flow
Logging in FLEXT is designed to support correlation ID tracking for distributed operations. This allows for the tracing of a single transaction across multiple stages of a pipeline, including parallel validation and analysis stages.
The diagram shows how correlation and operation identifiers are propagated through pipeline stages to ensure consistent logging context.
Sources: examples/advanced_processing_example.py:100-115, AGENTS.md:320-330, examples/README.md:95-105
Key Components and Data Structures
The logging system interacts with several Pydantic-based data structures that capture operational metrics and execution results. These models ensure that log data is typed and validated before being emitted.
Operational Result Models
| Model | Field | Description |
|---|---|---|
ProcessingResult | operation_id | Unique operation identifier for log correlation. |
ProcessingResult | processing_time | Latency metric for performance analysis. |
ValidationResult | item_id | Identifier for the specific item being validated. |
ValidationResult | violations | List of validation failures for detailed error logging. |
Sources: examples/advanced_processing_example.py:87-124
Metric Tracking in Pipelines
FLEXT utilizes PipelineStageData to maintain state and context across pipeline executions. This container allows the logger to capture the current state of "values" and "metadata" at any point in the lifecycle.
class PipelineStageData(m.BaseModel):
"""Data container for pipeline stage processing."""
class PipelinePayload(m.BaseModel):
values: t.JsonMapping = u.Field(default_factory=_new_data_value_map)
data: PipelinePayload = u.Field(
default_factory=lambda: PipelineStageData.PipelinePayload(values={}),
)
Sources: examples/advanced_processing_example.py:59-75
Integration with Parallel Processing
A critical aspect of FLEXT logging is maintaining context during concurrent execution. The platform extensively uses ThreadPoolExecutor for parallel batch processing. The logging system must handle asynchronous emissions and ensure that thread-local or passed context (like operation_id) is correctly associated with each log entry.
Parallel Execution Flow
The sequence demonstrates how context (Operation ID) is passed to workers and utilized during logging emissions.
Sources: examples/advanced_processing_example.py:202-225, examples/README.md:45-55
Configuration and Environment
Logging behavior is influenced by the platform's environment settings. FLEXT prohibits the raw usage of os.environ in production code, favoring settings abstractions that can configure logging levels, debug modes, and output formats.
| Setting | Usage | Source |
|---|---|---|
FLEXT_DEBUG_MODE | Enables verbose logging and trace output. | docker/docker-compose.oracle-wms.yml:32 |
PYTHONPATH | Ensures core logging modules are in path. | docker/docker-compose.oracle-wms.yml:29 |
ORACLE_WMS_ENABLE_REQUEST_LOGGING | Specifically enables API request/response logging. | docker/docker-compose.oracle-wms.yml:38 |
Sources: docker/docker-compose.oracle-wms.yml:20-40, AGENTS.md:335-340
Implementation Standards
According to FLEXT Canonical Engineering Law, logging must follow specific "Non-negotiables":
- Truth with Evidence: Claims in logs must include exact context and decisive output.
- Zero Bypass: Developers must use the project facades (e.g.,
ufor utilities,mfor models) rather than ad-hoc logging configurations. - Correlation Tracking: Distributed operations must include correlation IDs to prevent fragmented observability.
Sources: AGENTS.md:18-25, AGENTS.md:320-330, libs/init.py:10-35
Summary
Structlog Context-Aware Logging in FLEXT is not merely a diagnostic tool but a fundamental component of the platform's reliability architecture. By enforcing structured data models for results and utilizing correlation IDs across parallel pipelines, the system ensures that every automated action is backed by traceable, typed evidence. This supports the mission of maintaining a "Continuous Green" state across all enterprise data integration workloads.