Data Pipelines and ETL
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Data Pipelines and ETL (Extract, Transform, Load) system within the TAMV Online project serves as the backbone for synchronizing complex transportation data across the platform. This system is designed to handle high volumes of real-time data from various sources, ensuring that the "ambitious human-created system" maintains data integrity and availability for end-users and administrative modules.
The primary scope of these pipelines is to bridge the gap between raw data ingestion and the structured storage required for the system's operation. By automating the movement and transformation of data, the system reduces manual overhead and provides a consistent state for the TAMV Online ecosystem.
Sources: README.md:1-3, Arquitectura_Base.md
Data Pipeline Architecture
The architecture follows a modular approach where data flows from external or internal triggers through a transformation layer before reaching the final data store. This ensures that any data entering the system is validated and formatted according to the project's strict schema requirements.
Component Overview
The ETL process is divided into three distinct phases to ensure scalability and maintainability:
- Extraction: Captures raw data from API endpoints or direct database inputs.
- Transformation: Normalizes data formats, applies business logic, and filters out redundant information.
- Loading: Commits the processed data to the primary storage volumes.
The diagram above illustrates the sequential flow of data from ingestion to persistent storage.
Sources: Flujos_de_sistema.md, Estructura_de_datos.md
ETL Process Details
Extraction Phase
The extraction phase identifies and retrieves data from various system inputs. In TAMV Online, this primarily involves monitoring specific endpoints that receive updates from transit monitors or manual administrative entries.
| Parameter | Type | Description |
|---|---|---|
source_id | String | Unique identifier of the data origin. |
raw_payload | JSON | The unprocessed data packet received by the system. |
timestamp | DateTime | The exact moment the data was captured. |
Sources: Endpoints_API.md, Estructura_de_datos.md
Transformation Phase
During transformation, the system maps raw fields to the internal data model. This includes converting date formats, calculating derived values, and ensuring that relationships between entities (such as routes and vehicles) are maintained.
This sequence shows the internal verification steps taken before any data is considered "clean" for the system.
Sources: Flujos_de_sistema.md, Arquitectura_Base.md
Loading Phase
The loading phase interacts with the database layer to perform UPSERT operations. This ensures that existing records are updated with the latest information while new records are created without causing duplication.
Key data structures involved in the loading phase include:
- Vehicle Records: Updates on location and status.
- Route Definitions: Changes to paths or stops.
- User Logs: Historical tracking of system interactions.
Sources: Estructura_de_datos.md, Endpoints_API.md
System Synchronization
To maintain the "total documentation" standard, every ETL execution is logged. This provides a clear audit trail for debugging and system monitoring. The synchronization happens in near real-time to support the "Online" nature of the TAMV project.
| Feature | Description | File Reference |
|---|---|---|
| Real-time Sync | Immediate processing of incoming data packets. | Flujos_de_sistema.md |
| Schema Enforcement | Rejection of data that does not meet the structural requirements. | Estructura_de_datos.md |
| Audit Logging | Every transformation step is recorded for system integrity. | Arquitectura_Base.md |
Sources: Flujos_de_sistema.md, Estructura_de_datos.md, Arquitectura_Base.md
Conclusion
The Data Pipelines and ETL system is fundamental to the TAMV Online project, providing the necessary infrastructure to manage complex data flows reliably. By utilizing a structured ETL approach, the system ensures that the ambitious goals of the project are met with high-quality, synchronized data across all modules.