E2E & Docker Test Setup
Relevant source files
The following files were used as context for generating this wiki page:
The FLEXT Meltano project utilizes a comprehensive End-to-End (E2E) and Docker-based testing infrastructure to validate the "Go ↔ Python ↔ Meltano" bridge architecture. This setup ensures that data integration pipelines function correctly within production-like environments, utilizing actual database services and network communication.
The testing scope encompasses the entire journey from service requests through bridge execution to final data output, ensuring data integrity and error recovery across the full FLEXT ecosystem.
Sources: tests/e2e/README.md:14-22, tests/README.md:16-24
🏗️ Docker Infrastructure Architecture
The testing environment relies on a containerized stack to provide isolated, deterministic dependencies for integration and E2E tests. This stack is primarily managed via docker-compose configurations.
Core Service Components
The infrastructure consists of three primary services required for a full pipeline execution:
- PostgreSQL: Serves as either a source (
tap) or target for data operations, as well as a state store for Meltano. - Redis: Utilized for caching and message brokerage within the bridge architecture.
- Meltano: The core orchestration engine running within a specialized test environment.
The diagram above illustrates the interaction between the test runner on the host machine and the containerized services.
Sources: tests/e2e/README.md:118-138, tests/integration/test_docker_integration.py:84-126
Environment Configuration
Configuration is managed through specific environment variables and Docker Compose files.
| Service | Default Port | E2E Environment Variable | Description |
|---|---|---|---|
| PostgreSQL | 5434 | POSTGRES_URL | Test database for e2e validation |
| Redis | 6381 | REDIS_URL | Cache service for test state |
| Meltano | Variable | MELTANO_PROJECT_ROOT | Path to the test Meltano project |
Sources: tests/e2e/README.md:140-149, tests/unit/fixtures.py:166-193
🧪 Testing Categories and Lifecycle
The project organizes tests into specific categories that utilize Docker resources with varying levels of intensity.
Integration Tests (Docker-Based)
Integration tests verify component interactions with real external dependencies. These tests check service availability and basic operations like SQL execution and Redis key-value persistence.
- Classes:
TestsFlextMeltanoDockerIntegrationhandles connectivity and health checks. - Methods:
_assert_postgres_operationsand_assert_redis_operationsvalidate canonical data flows.
Sources: tests/integration/test_docker_integration.py:21-70, tests/README.md:42-46
End-to-End (E2E) Tests
E2E tests validate complete user journeys. These are marked with @pytest.mark.e2e and usually require a full production-like stack.
This sequence shows the high-level flow of an E2E pipeline test from initialization to output validation.
Sources: tests/e2e/README.md:162-184, tests/unit/fixtures.py:38-55
🔧 Test Fixtures and Lifecycle Management
Lifecycle management is handled through a docker_manager fixture which abstracts the starting and stopping of containers.
Docker Manager Logic
The docker_manager (using tk.stack) is responsible for ensuring the test stack is healthy before test execution begins. It includes retry logic for transient errors, such as database startup delays.
- Setup:
docker_manager.execute()brings up the stack defined in the compose file. - Health Checks:
docker_services.ready(port=...)verifies port availability before proceeding. - Teardown:
docker_manager.down()ensures containers are removed to maintain test isolation.
Sources: tests/unit/fixtures.py:126-146, tests/integration/test_docker_integration.py:128-142
Fixture Categories
| Category | Description | Examples |
|---|---|---|
| Data Fixtures | Sample datasets and schemas | sample_csv_data, user_schema_fixture |
| Service Fixtures | Container and environment setup | postgres_service, redis_service |
| Pipeline Fixtures | Meltano project templates | meltano_test_project, singer_state |
Sources: tests/fixtures/README.md:14-35, tests/unit/fixtures.py:166-193
📊 Performance and Quality Standards
The E2E suite follows strict performance and resource thresholds to ensure efficiency in CI/CD pipelines.
- Execution Time: Simple pipelines must complete in < 2 minutes, while the full E2E suite is expected to take < 2 hours.
- Resource Limits: Total memory usage should remain below 4GB for the environment.
- Quality Gates: A minimum of 90% line coverage is required for core modules, and zero flaky tests are tolerated in the CI/CD environment.
Sources: tests/e2e/README.md:151-160, tests/README.md:131-136, pyproject.toml:210-213
📋 Summary
The E2E and Docker test setup in FLEXT Meltano provides a robust framework for validating complex data integration workflows. By utilizing docker-compose to orchestrate PostgreSQL, Redis, and Meltano, the project ensures that integration and end-to-end tests are conducted in environments that closely mirror production, guaranteeing high reliability for the Go ↔ Python bridge architecture.