Architecture Baseline
Relevant source files
The following files were used as context for generating this wiki page:
FLEXT is an enterprise-grade, multi-package Python workspace designed for data integration, platform tooling, and operational connectors. The architecture is centered around a typed foundation (flext-core) that enforces architectural invariants across a suite of domain-specific and infrastructure packages. The primary goal of the FLEXT platform is to ensure that integration logic is built upon reusable, validated, and strictly typed primitives.
The current architecture baseline (v0.13.0) emphasizes a "Railway Pattern" for error handling, pervasive parallel processing for performance, and a "Single Source of Truth" (SSOT) philosophy for configuration and documentation. It employs a managed monorepo structure where all flext-* packages inherit governance from a central core, ensuring zero ad-hoc code through the use of canonical facades and fallible path contracts.
Sources: README.md:3-10, AGENTS.md:143-150
Core Architectural Patterns
The Railway Pattern
FLEXT utilizes a functional "Railway Pattern" for error handling, primarily through the r[T] (Result) type. This approach ensures that pipelines are failure-resistant; operations return a Result object that either contains the success value or failure context. Execution typically stops on the first failure in a chain, preventing exception propagation and ensuring predictable state transitions.
Sources: examples/README.md:65-70, AGENTS.md:167-168
Parallel Execution and Batching
The platform is designed for high-throughput data processing using ThreadPoolExecutor for concurrent operations. The architecture supports configurable worker thread pools and batch processing to optimize memory efficiency and CPU utilization during heavy integration workloads.
Sources: examples/README.md:72-77, examples/advanced_processing_example.py:168-185
The diagram illustrates the standard pipeline flow from input through parallel stages, adhering to the railway pattern's fail-fast logic.
Sources: examples/advanced_processing_example.py:117-145, examples/complete_workflow_example.py:195-235
Workspace Structure and Governance
The workspace is a managed monorepo governed by a strict hierarchy of engineering laws codified in AGENTS.md. This governance chain ensures that every package follows identical naming conventions, contract styles, and runtime constraints.
| Component | Responsibility | Relevant Packages |
|---|---|---|
| Platform Core | Foundational typed primitives and MRO classes | flext-core, flext-infra, flext-tests |
| Capabilities | Common services like CLI, API, Auth, and Observability | flext-cli, flext-api, flext-auth, flext-grpc |
| Domain Packages | Logic for specific protocols/systems (LDAP, Oracle) | flext-ldap, flext-db-oracle, flext-oracle-wms |
| Integrations | Connector logic (Tap, Target, dbt) | flext-tap-*, flext-target-*, flext-dbt-* |
Sources: README.md:17-23, AGENTS.md:143-152
Package Metadata and Managed Files
The project uses pyproject.toml as the SSOT for metadata. Specific sections are marked as [MANAGED] and enforced by flext_infra.deps.modernizer. This includes the build system (hatchling), development dependencies, and tool configurations for linting (Ruff), type checking (Mypy, Pyright), and testing (Pytest).
Sources: pyproject.toml:1-15, README.md:31-36
Data Flow and Processing Architecture
Processing in FLEXT is organized into declarative pipelines. A pipeline consists of multiple stages (e.g., validate, process, analyze) that transform data objects derived from Pydantic models.
Orchestration Logic
The WorkflowOrchestrator manages the lifecycle of a processing task, including context setup with correlation IDs, resource management, and metrics aggregation.
Sequence of a workflow execution managing context, parallel execution, and final metrics aggregation.
Sources: examples/complete_workflow_example.py:122-145, examples/advanced_processing_example.py:117-130
Configuration Management
The architecture prohibits hardcoded values. Every configurable fact must reside in config/*.yaml or typed settings objects. Tests validate these configurations by construction, ensuring that a change in configuration does not break the test suite unless the contract itself is violated.
Sources: AGENTS.md:14-25, AGENTS.md:176-180
Docker Infrastructure and Persistence
FLEXT employs a centralized Docker management system called tk to handle container lifecycles during testing and development. All Docker artifacts are consolidated to prevent duplication across sub-packages.
Centralized Management
All Dockerfiles are stored in docker/images/, and docker-compose files are managed centrally. Individual project directories are prohibited from containing local Docker configurations.
| Container Type | Port | Purpose |
|---|---|---|
| Standard Oracle DB | 1522 | Standard DB testing for flext-db-oracle |
| FLEXT OUD | 3389 | OpenLDAP simulating Oracle Unified Directory |
| Generic OpenLDAP | 3390 | General LDAP/LDIF testing |
Sources: docker/README.md:85-115
Container Lifecycle
The tk manager ensures:
- Test Isolation: Clean container state for each test.
- Automatic Cleanup: Resources are reclaimed after test completion.
- Dirty State Management: Compromised containers are marked "dirty" and recreated from scratch on the next run.
Sources: docker/README.md:147-165
Development and Quality Gates
The architecture baseline mandates "Continuous Green" status. No code is considered "Done" until it passes a comprehensive suite of quality gates.
Mandatory Verification Steps
- Static Analysis: Ruff (linting/formatting), Pyright/Mypy (strict typing), and Pyrefly.
- Evidence-Based Truth: Claims of completion require exact command output and exit status.
- Documentation: API references are generated from code docstrings (the SSOT), and architectural prose must be updated in the same PR as the code change.
Sources: AGENTS.md:37-45, AGENTS.md:65-70, README.md:31-36
Naming Conventions
The project uses canonical aliases for internal facades:
c,m,t,p,u: Standard project facades.r,e,s,x: Operational/Result aliases.
Sources: AGENTS.md:166-170, examples/advanced_processing_example.py:43-45
Summary
The FLEXT Architecture Baseline provides a rigid yet scalable framework for enterprise data integration. By enforcing a unified governance model across a monorepo, the platform ensures high reliability through the Railway Pattern, high performance through managed parallel processing, and strict operational consistency through a centralized infrastructure and configuration management system. All developments are anchored by the "North Star" goal of guaranteed typed and validated integration primitives.
Sources: AGENTS.md:143-150