Testing Quantitative Models
Relevant source files
The following files were used as context for generating this wiki page:
Testing quantitative models within the hyperdrive library involves a multi-layered approach that includes mathematical validation, historical backtesting, and machine learning classifier evaluation. The system relies on the Historian class to generate signals and portfolios, which are then analyzed using metrics such as the Sortino Ratio, Sharpe Ratio, and Profit Factor.
The testing framework ensures that trading strategies—ranging from simple Buy and Hold to complex machine learning-derived portfolios—are statistically sound and performant against historical OHLC (Open, High, Low, Close) market data. This process integrates data preprocessing via the Calculator and temporal management via the TimeTraveller.
Sources: README.md:95-125, hyperdrive/History.py:22-37
Core Testing Components
Portfolio Backtesting Logic
The primary mechanism for testing quantitative models is through portfolio simulation. The Historian class leverages vectorbt to transform price series and signals into measurable portfolios.
- Buy and Hold Strategy: Acts as a benchmark by assuming a long position from the start of the dataset.
- Signal-Based Strategy: Executes trades based on boolean signal arrays (True for buy/long, False for sell/short).
- Random Strategy Generation: Used to establish a baseline of "lucky" performance by generating thousands of random signal sets and filtering for the top performers based on Sortino and Calmar ratios.
Sources: hyperdrive/History.py:31-43, hyperdrive/History.py:77-98, scripts/update_api.py:39-46
Backtesting Data Flow
The following diagram illustrates how market data is processed through the Historian to produce performance metrics.
This diagram represents the flow from raw data to comparative performance analysis.
Sources: README.md:105-130, scripts/update_api.py:32-60
Quantitative Analysis Tools
Mathematical Preprocessing
The Calculator class provides essential mathematical functions used to derive features for models or validate data integrity.
| Function | Purpose | Example Usage |
|---|---|---|
delta | Calculates percentage change between periods | Identifying price momentum |
derive | Calculates the rate of change in a series | Trend strength analysis |
smooth | Applies smoothing algorithms to reduce noise | Signal filtering |
cv | Calculates Coefficient of Variation | Volatility assessment |
Sources: test/test_Calculus.py:11-39, hyperdrive/History.py:23
Temporal Validation
Testing requires precise alignment of dates and timeframes. The TimeTraveller ensures that historical ranges are valid and that look-ahead bias is avoided by converting human-readable strings (e.g., '1y', '3m') into specific datetime objects.
Sources: test/test_TimeMachine.py:14-35
Machine Learning Evaluation
The project includes a robust pipeline for testing predictive models using various classifiers. This involves preprocessing data to ensure the testing environment reflects realistic trading conditions.
Undersampling and Scaling
To prevent model bias, particularly in imbalanced financial datasets, the Historian implements an undersample method. This balances the training set so that "True" and "False" signals are represented equally before being passed to classifiers.
Sources: hyperdrive/History.py:104-128, test/test_History.py:53-55
Classifier Performance Testing
Models are evaluated using a suite of algorithms. The run_classifiers method fits data to multiple models and filters them based on their score and a "ratio" check to prevent overfitting (ensuring the training-to-test score ratio is below 1.15).
This sequence shows the iterative process of training and validating multiple quantitative models simultaneously.
Sources: hyperdrive/History.py:149-178, test/test_History.py:57-61
Supported Classifiers
The system evaluates quantitative models using the following standard algorithms:
- Nearest Neighbors & Naive Bayes
- Support Vector Machines (Linear and RBF Kernels)
- Decision Trees & Random Forests
- Multi-layer Perceptron (Neural Net)
- AdaBoost & Quadratic Discriminant Analysis (QDA)
Sources: hyperdrive/History.py:150-164
Performance Metrics Summary
When testing a quantitative model, the system generates a standard report of metrics to determine if a strategy is viable.
| Metric | Definition / Source |
|---|---|
| Total Return [%] | The cumulative percentage gain of the portfolio. |
| Max Drawdown [%] | The largest peak-to-trough decline. |
| Sharpe Ratio | Risk-adjusted return based on volatility. |
| Sortino Ratio | Risk-adjusted return focusing on downside volatility. |
| Win Rate [%] | The percentage of trades that resulted in a profit. |
| Profit Factor | The ratio of gross profits to gross losses. |
Sources: README.md:115-125, scripts/update_api.py:32-40
Testing quantitative models in this project is a rigorous process of converting raw market data into backtested portfolios and validated machine learning models. By utilizing standardized metrics and avoiding overfitting through strict score ratio checks, the system provides a framework for identifying high-probability trading strategies.