ML Models & Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The ML Models & Pipeline in AlphaPy represents the core execution engine of the framework, designed to automate the end-to-end machine learning workflow. It encompasses data ingestion, feature engineering, model selection, optimization, and evaluation for both classification and regression tasks. The pipeline is architected to support multiple algorithmic frameworks, including scikit-learn, Keras, XGBoost, LightGBM, and CatBoost, allowing for comparative analysis and the creation of ensemble models.
The pipeline operates in two primary modes: Training Mode, which involves fitting models, performing grid searches, and generating performance metrics; and Predict Mode, which loads serialized model predictors to generate results on new datasets. This infrastructure serves as the foundation for specialized flows like MarketFlow for financial markets and SportFlow for athletic event prediction.
Sources: README.rst:11-20, alphapy/main.py:67-73, alphapy/model.py:65-73
Architecture and Core Components
The pipeline is centered around the Model class, which acts as a unified container for data matrices, algorithm specifications, estimators, and results.
Model Object Structure
The Model class (in alphapy/model.py) maintains the state of the machine learning process. It stores training and testing partitions (X_train, X_test, y_train, y_test), a list of algorithms to execute (algolist), and dictionaries for storing serialized estimators, feature importances, and predictions.
Sources: alphapy/model.py:75-125
The diagram above illustrates the relationship between the central Model container and the individual Estimator objects configured for the pipeline.
Sources: alphapy/model.py:85-136, alphapy/estimators.py:107-130
Pipeline Execution Flow
The AlphaPy pipeline follows a structured sequence of operations defined in training_pipeline and prediction_pipeline.
Training Pipeline Logic
- Data Loading & Splitting: Retrieves data partitions. If no test data is found, it splits the training data based on the
splitparameter. - Feature Engineering: Applies transformations, drops specified features, and creates interactions or categorical crosstabs.
- Preprocessing: Performs low-variance feature removal and optional over/under-sampling for classification.
- Estimator Initialization: Retrieves estimators based on the
algos.ymlconfiguration. - Model Fitting & Optimization:
first_fit: Initial training and cross-validation.rfecv_search: Recursive Feature Elimination (optional).hyper_grid_search: Hyperparameter optimization (optional).
- Ensembling: Creates a
BLENDmodel if multiple algorithms are used. - Evaluation: Generates metrics and plots (ROC, Confusion Matrix, etc.).
- Persistence: Saves the "BEST" model and predictions to disk.
Sources: alphapy/main.py:75-220
The flowchart describes the sequential stages of the AlphaPy training pipeline as implemented in the training_pipeline function.
Sources: alphapy/main.py:75-220, alphapy/model.py:465-480
Estimator Management
AlphaPy utilizes an internal mapping system to manage diverse model types. The Estimator class wraps scikit-learn compatible functions, including deep learning models via Keras wrappers.
Supported Algorithms
| Abbreviation | Algorithm | Type |
|---|---|---|
RF / RFR | Random Forest | Classifier / Regressor |
XGB / XGBR | XGBoost | Classifier / Regressor |
KERASC / KERASR | Keras | Sequential Deep Learning |
LOGR / LR | Logistic / Linear Regression | Linear Models |
SVM / LSVM | Support Vector Machine | Kernel-based Models |
LGB / LGBR | LightGBM | Gradient Boosting |
CATB / CATBR | CatBoost | Gradient Boosting |
| Sources: alphapy/estimators.py:141-185 |
Blending and Ensembling
When multiple algorithms are executed, AlphaPy generates a BLEND model. For classification, this is a LogisticRegression meta-estimator that uses the probabilities of base models as inputs. For regression, a RidgeCV meta-estimator is used.
Sources: alphapy/model.py:657-735
Scoring and Evaluation
The system supports a wide array of metrics mapped to specific ModelType objectives (Maximize or Minimize).
| Metric | Model Type | Objective |
|---|---|---|
roc_auc | Classification | Maximize |
neg_log_loss | Classification | Minimize |
accuracy | Classification | Maximize |
r2 | Regression | Maximize |
neg_mean_squared_error | Regression | Minimize |
| Sources: alphapy/estimators.py:53-74, alphapy/globals.py:165-177 |
Metric Generation Logic
The generate_metrics function attempts to calculate every available metric from scikit-learn for the given model type. It uses a "brute-force" approach, catching exceptions to ensure the pipeline continues even if specific metrics are not applicable to the data.
Sources: alphapy/model.py:743-847
Model Persistence and Prediction
AlphaPy uses joblib for serializing standard estimators and .h5 files for Keras models.
Persistence Components
- Predictor: The best estimator saved as
model_[timestamp].pklor.h5. - Feature Map: Contains information for transforming new data, including univariate and RFE support vectors, saved as
feature_map_[timestamp].pkl. - Predictions: Outputs saved as
predictions,probabilities, andrankingsin the output directory.
Sources: alphapy/model.py:284-386, alphapy/model.py:925-1025
The sequence diagram details the process of restoring a model state for inference during the prediction pipeline execution.
Sources: alphapy/model.py:284-338, alphapy/main.py:228-293
Summary
The ML Models & Pipeline module in AlphaPy provides a robust, configuration-driven framework for automated machine learning. By abstracting the complexities of model selection, hyperparameter tuning, and ensemble creation into a unified Model class and structured pipelines, it enables consistent and reproducible experimentation across financial and sporting domains. The reliance on standard formats like YAML for configuration and joblib for persistence ensures that models can be easily moved from training environments to production forecasting.
Sources: README.rst:11-30, alphapy/main.py:317-375