General Utilities
Relevant source files
The following files were used as context for generating this wiki page:
General Utilities in AlphaPy comprise a suite of helper functions, global constants, and standardized formatting tools that support the broader machine learning pipeline. These utilities facilitate common operations such as date manipulation, file system interactions, data persistence, and list management, ensuring consistency across different modules like MarketFlow and SportFlow.
The scope of these utilities extends from low-level string delimiters and numerical constants to high-level functions for managing model predictors and feature maps. By centralizing these operations, AlphaPy maintains a clean architecture where specialized modules can rely on robust, tested shared logic for administrative and data-handling tasks.
Sources: alphapy/utilities.py, alphapy/globals.py
Date and Time Management
AlphaPy relies heavily on standardized date formats, specifically YYYY-mm-dd, to synchronize training and prediction phases across financial and sports datasets.
Core Date Functions
The system provides utilities to validate date strings, calculate offsets, and generate datestamps for file versioning.
get_datestamp: Returns the current date inYYYYmmddformat, primarily used for naming output files and model checkpoints.valid_date: A validation wrapper that ensures input strings match the%Y-%m-%dformat, raising aValueErrorif the check fails.subtract_days: Adjusts a given date string by a specified number of days, returning a new valid date string.
Sources: alphapy/utilities.py:44-142
Date Logic Flow
The following diagram illustrates how AlphaPy validates and manipulates dates during the pipeline execution.
The diagram shows the validation path for date strings and subsequent manipulation via offsets.
Sources: alphapy/utilities.py:104-142
File System and Data Persistence
The project utilizes a specific directory structure for organizing input data, models, and outputs. Utilities manage file discovery and the saving of structured data.
Persistence Utilities
| Function | Purpose | Key Parameters |
|---|---|---|
most_recent_file | Locates the latest file matching a wildcard pattern based on creation time. | directory, file_spec |
np_store_data | Saves NumPy arrays to text files with specific delimiters and extensions. | data, dir_name, file_name, extension, separator |
save_predictor | Persists a trained model (Pickle or H5) with a timestamp. | model, timestamp |
load_predictor | Discovery and loading of the most recent model file. | directory |
Sources: alphapy/utilities.py:65-98, alphapy/model.py:461-512
Model and Feature Persistence Flow
AlphaPy manages model state by saving both the estimator and the associated feature map to ensure reproducibility during the prediction phase.
This diagram depicts the flow of saving model state during training and retrieving the most recent version for predictions.
Sources: alphapy/model.py:461-555, alphapy/utilities.py:65-81
Global Constants and Delimiters
Standardization is maintained through a set of global variables defined in globals.py. These constants are used across the entire package to ensure uniform file paths and data separation.
Delimiters and Numerical Constants
- Path/String Delimiters:
SSEP:/(System Separator)PSEP:.(Period Separator)USEP:_(Underscore Separator)BSEP:
- Quartiles:
Q1(0.25),Q2(0.50),Q3(0.75) are used for statistical plotting and facet grid calculations.
Sources: alphapy/globals.py:34-56, alphapy/plots.py:734-740
List and Data Structure Manipulation
Utility functions handle specialized operations on Python lists and identifiers, often used for feature set pruning.
remove_list_items: Removes a list of elements from another list, returning the modified sublist. This is frequently used inplots.pyand feature selection.valid_name: Uses a regex pattern (r"^[^\d\W]\w*\Z") to ensure that strings are valid alphanumeric identifiers (not starting with a digit).
Sources: alphapy/utilities.py:148-185
Specialized Application Utilities
Different "Flows" within AlphaPy (Market and Sport) utilize general utilities to bridge raw data with machine learning models.
Sports-Specific Logic
In sport_flow.py, general list and date logic is applied to generate team-specific features.
get_streak: Calculates Boolean streaks within a series (e.g., winning/losing streaks).add_features: A utility that initializes new columns in a DataFrame based on a dictionary of types.
Sources: alphapy/sport_flow.py:270-323
Plotting Utilities
The plots.py module uses get_plot_directory to standardize output paths by joining the model directory with a plots sub-folder using the global SSEP.
write_plot: A multi-library utility that handles saving or displaying plots frommatplotlib,seaborn, orbokeh.
Sources: alphapy/plots.py:165-245
Summary
General Utilities provide the foundational infrastructure for AlphaPy. By centralizing date validation, file persistence logic, and global constants, the framework ensures that complex pipelines like MarketFlow and SportFlow operate on a consistent and predictable environment. This modular approach allows for robust error handling and standardized data output across all machine learning tasks.