History & Historian Module
Relevant source files
The following files were used as context for generating this wiki page:
The History & Historian module is a core component of the hyperdrive library, specializing in quantitative research, backtesting, and machine learning for algorithmic trading. It bridges the gap between raw market data and actionable trading strategies by providing tools to generate signals, evaluate portfolios, and train predictive models.
The module primarily revolves around the Historian class, which leverages the vectorbt library for performance analysis and scikit-learn for classification tasks. It allows developers to test "Buy and Hold" strategies, generate optimal historical signals, and perform advanced data preprocessing such as undersampling and Principal Component Analysis (PCA).
Sources: README.md:12-16, hyperdrive/History.py:1-26
Core Architecture and Components
The Historian class serves as the main entry point for historical analysis. It maintains a relationship with a Calculator instance for mathematical operations and provides a suite of methods for strategy simulation.
Class Relationship
The following diagram illustrates the primary classes involved in the historical analysis workflow:
Portfolio Management and Backtesting
The module provides high-level abstractions for simulating trading strategies. It distinguishes between benchmark strategies (Buy and Hold) and signal-driven strategies.
Backtesting Workflow
The process typically involves fetching market data via MarketData and passing it to the Historian to generate a vectorbt.Portfolio object.
| Method | Description | Key Parameters |
|---|---|---|
buy_and_hold | Creates a portfolio based on a passive holding strategy. | close (Prices), init_cash (Default: 1000) |
create_portfolio | Creates a portfolio based on boolean entry/exit signals. | close, signals, fee (Default: 0) |
get_optimal_signals | Identifies historical local minima/maxima to find "perfect" signals. | close, n (Order window), method (fill type) |
Sources: hyperdrive/History.py:32-44, hyperdrive/History.py:61-75, scripts/update_api.py:46-56
Signal Processing and Data Manipulation
Signal manipulation ensures that historical data is continuous and usable for machine learning models.
Signal Filling Methods
Historian provides logic to handle missing data or sparse signals:
- Fill: Uses forward-filling (
ffill) or nearest-neighbor interpolation to convert sparse arrays into continuous boolean signals. - Unfill: A utility that replaces consecutive identical values with
None(except the first instance), effectively identifying the discrete moments of signal changes (entry/exit points).
Sources: hyperdrive/History.py:46-60, test/test_History.py:27-37
Machine Learning Integration
A significant portion of the module is dedicated to preparing market features for classification models and evaluating their performance.
Data Preprocessing Pipeline
The undersample method performs a comprehensive transformation of features ($X$) and labels ($y$):
- Drop Missing Values: Removes rows with
NaNin either features or labels. - Balancing: Undersamples the majority class (usually "hold" or "False" signals) to match the count of the minority class, ensuring a 50/50 distribution for training.
- Standardization: Applies
StandardScalerto normalize feature scales. - Dimensionality Reduction: Utilizes Principal Component Analysis (PCA) to reduce feature space while retaining variance.
Sources: hyperdrive/History.py:100-134
Classifier Evaluation
The run_classifiers method iterates through multiple scikit-learn algorithms to identify the most effective model for the provided data.
| Classifier Category | Specific Algorithms Used |
|---|---|
| Linear/Kernel | Linear SVM, RBF SVM, Naive Bayes |
| Tree/Ensemble | Decision Tree, Random Forest, AdaBoost |
| Discriminant | QDA |
| Instance-based | K-Nearest Neighbors |
| Neural | MLPClassifier |
The method applies a specific filter: it only returns models where the ratio of training score to test score is less than 1.15, effectively pruning models that are heavily overfit.
Sources: hyperdrive/History.py:155-186
Strategy Evaluation Metrics
The module is used to calculate various quantitative metrics to compare strategies against benchmarks. The scripts/update_api.py utilizes these for production reporting.
| Metric | Context |
|---|---|
| Total Return [%] | Overall percentage gain/loss of the portfolio. |
| Max Drawdown [%] | The largest peak-to-trough decline. |
| Profit Factor | Ratio of gross profit to gross loss. |
| Sharpe/Sortino Ratio | Risk-adjusted return measures. |
| Win Rate [%] | Percentage of trades that were profitable. |
Sources: scripts/update_api.py:40-45, README.md:95-108
Summary
The History & Historian module provides a robust framework for quantitative analysis within hyperdrive. By integrating backtesting (via vectorbt) with a sophisticated machine learning pipeline (via scikit-learn), it enables the transition from historical data exploration to the validation of predictive trading models. Its focus on balanced training sets and dimensionality reduction addresses common pitfalls in financial data science, such as class imbalance and overfitting.