Data Loading & Processing
Relevant source files
The following files were used as context for generating this wiki page:
Data Loading & Processing in AlphaPy serves as the foundational layer for transforming raw data into a format suitable for machine learning models. The system handles diverse data sources, including market data and sports statistics, utilizing pandas for internal data structures and file operations. The pipeline is designed to manage three primary data partitions: train, test, and predict, ensuring consistency across the modeling lifecycle.
The architecture emphasizes a "Flow" pattern (e.g., MarketFlow and SportFlow) where domain-specific data is ingested, augmented with engineered features, and then saved into unified "input" files for the core AlphaPy training or prediction pipelines.
Data Flow Architecture
The data lifecycle begins with raw file ingestion or API retrieval, followed by partitioning and feature sequencing. Data is typically moved from a data directory to an input directory after initial processing.
The diagram shows the high-level movement from raw sources through domain-specific flows into the final training pipeline.
Sources: alphapy/market_flow.py:171-200, alphapy/sport_flow.py:440-470, alphapy/main.py:100-150
Frame Management and I/O
AlphaPy utilizes a Frame class to track DataFrames in memory and a set of utility functions to handle persistent storage. The system relies heavily on CSV formatted files with configurable separators and extensions.
Key I/O Functions
| Function | Purpose | Key Parameters |
|---|---|---|
read_frame | Loads a CSV file into a pandas.DataFrame. | directory, filename, extension, separator |
write_frame | Saves a DataFrame to disk. | df, directory, filename, index_label |
load_frames | Reads a group of related frames (e.g., multiple stock symbols) into memory. | group, splits |
sequence_frame | Creates sequences of lagging and leading values for time-series analysis. | target, forecast_period, lag_period |
Sources: alphapy/frame.py:108-164, alphapy/frame.py:255-296
Sequencing Logic
For time-series data, the sequence_frame function is critical. It shifts columns to create historical "lags" and forward-looking "targets."
# Illustration of lag/lead creation
for i in range(lag_period, 0, -1):
new_cols.append(df[df_cols].shift(i))
new_cols.append(pd.DataFrame(df[target].shift(1-forecast_period)))
Sources: alphapy/frame.py:284-293
Analysis and Partitioning
Data is split into Partition.train, Partition.test, and Partition.predict. If a dedicated test file is missing during training, AlphaPy automatically splits the training data based on a user-defined split ratio.
The Analysis Class
The Analysis class encapsulates a group of data (e.g., a set of stocks) and a model specification. The run_analysis function manages the transition from raw historical frames to the final train/test/predict CSV files used by the estimator.
This sequence illustrates how group-based data is aggregated and partitioned before model training.
Sources: alphapy/analysis.py:91-160, alphapy/main.py:126-135
Domain-Specific Processing
MarketFlow Data Processing
MarketFlow fetches data based on a "Space" (subject, schema, fractal). It supports intraday offsets and applies technical indicators via vmapply before consolidating data for the model.
Sources: alphapy/market_flow.py:182-198
SportFlow Data Processing
SportFlow processes seasonal game data to generate team-specific statistics. It calculates rolling windows for point margins, streaks, and "delta" features (Home statistics minus Away statistics).
| Feature Type | Calculation Logic |
|---|---|
| Streaks | Consecutive wins/losses calculated via get_streak. |
| Deltas | home_stat - away_stat via generate_delta_data. |
| Rolling Avg | Calculated using pandas.rolling(window).mean(). |
Sources: alphapy/sport_flow.py:215-280, alphapy/sport_flow.py:415-425
Configuration Parameters
Data processing is controlled via the data section of model.yml.
| Parameter | Type | Description |
|---|---|---|
drop | list | Columns to be removed before training. |
target | str | The label or value the model is intended to predict. |
split | float | Percentage of training data to use for internal validation. |
sampling | bool | Whether to use over/under sampling for imbalanced classes. |
Sources: alphapy/model.py:245-270, alphapy/examples/NCAAB/config/model.yml:7-20
Summary
The Data Loading & Processing module ensures that raw data is systematically ingested, sequenced for time-series dependencies, and partitioned for robust model evaluation. By utilizing domain-specific flows like MarketFlow and SportFlow, AlphaPy standardizes complex feature engineering into a unified pipeline that feeds the training_pipeline and prediction_pipeline.