Optimization & Hyperparameter Tuning

Relevant source files

The following files were used as context for generating this wiki page:

• alphapy/model.py
• alphapy/main.py
• alphapy/estimators.py
• alphapy/analysis.py
• alphapy/examples/Kaggle/config/model.yml
• alphapy/optimize.py
• alphapy/space.py

Optimization and Hyperparameter Tuning in AlphaPy represent a critical phase of the machine learning pipeline where model performance is refined through systematic search and feature reduction. This process ensures that selected algorithms are not only trained on data but are also configured with the most effective parameters and feature subsets to maximize predictive accuracy or minimize loss.

The framework utilizes a combination of grid search, recursive feature elimination (RFE), and cross-validation to find the "Best" model. These operations are controlled via the model.yml configuration and executed within the training_pipeline of the main module.

Sources: alphapy/main.py:180-210, alphapy/model.py:644-670

Hyperparameter Search Architecture

AlphaPy supports hyperparameter optimization primarily through grid search. The training_pipeline iterates through a list of specified algorithms, performing an initial fit followed by an optional grid search if enabled in the specifications.

The Search Process

The optimization flow typically follows these steps:

1. First Fit: An initial model is fitted to establish a baseline score.
2. RFE (Optional): Recursive Feature Elimination is performed to prune less significant features.
3. Grid Search (Optional): If grid_search is true, the system invokes hyper_grid_search using the parameter grids defined in algos.yml.

This diagram illustrates the sequence of optimization steps applied to each algorithm in the model's algorithm list.
Sources: alphapy/main.py:228-255, alphapy/model.py:532-550

Configuration Parameters

Optimization behavior is governed by the model section of the model.yml configuration file. Key parameters determine how many folds are used for cross-validation and how the grid search is executed.

Parameter	Type	Description
grid_search	Boolean	Enables or disables hyperparameter grid search.
gs_iters	Integer	Number of iterations for randomized grid search.
gs_random	Boolean	If true, uses RandomizedSearchCV; otherwise uses GridSearchCV.
gs_sample	Boolean	Whether to subsample data for the grid search.
cv_folds	Integer	Number of cross-validation folds.
scoring_function	String	The metric used to evaluate model performance (e.g., 'roc_auc').

Sources: alphapy/model.py:220-245, alphapy/examples/Kaggle/config/model.yml:25-35

Feature Optimization and RFE

Recursive Feature Elimination (RFE) is a specific type of optimization that selects features by recursively considering smaller and smaller sets of features.

Implementation Details

• Initial Evaluation: The system checks if the estimator has coef_ or feature_importances_ attributes.
• Search: rfecv_search is called to find the optimal number of features using cross-validation.
• Support Vectors: The resulting support vector (the features kept) is stored in the model.feature_map to be reused during the prediction phase.

Sources: alphapy/model.py:730-740, alphapy/main.py:240-248

Scorer and Objective Functions

Tuning is driven by the scorer, which determines the Objective (maximize or minimize). AlphaPy maps scikit-learn metrics to these objectives to determine which model is "best."

Scorer	Model Type	Objective
accuracy	Classification	Maximize
neg_log_loss	Classification	Minimize
roc_auc	Classification	Maximize
r2	Regression	Maximize
neg_mean_squared_error	Regression	Minimize

Sources: alphapy/estimators.py:52-72, alphapy/model.py:685-700

Model Blending as Optimization

When multiple algorithms are selected, AlphaPy performs an additional optimization step called "Blending." This creates an ensemble model that combines the predictions of individual optimized models.

The blender uses LogisticRegression for classification and RidgeCV for regression to determine the optimal weights for each model's predictions.
Sources: alphapy/model.py:760-800, alphapy/main.py:260-265

Conclusion

The Optimization & Hyperparameter Tuning system in AlphaPy provides a robust framework for automated model improvement. By integrating scikit-learn's search capabilities with custom logic for feature selection and model blending, it allows developers to systematically explore parameter spaces and feature subsets, ensuring the final "BEST" model is highly optimized for the target objective.