Algorithm Testing Strategies
Relevant source files
The following files were used as context for generating this wiki page:
Algorithm testing in the Universal Portfolios project ensures the mathematical and logical integrity of Online Portfolio Selection (OLPS) strategies. The project employs a combination of unit testing via pytest, synthetic data generation for stress testing, and performance profiling to validate that algorithms maximize final wealth without introducing look-ahead bias or numerical instability.
The testing scope includes verifying weight calculation logic, transaction fee impact, and the stability of results across different asset subsets and timeframes. Key components involved in this process are the Algo base class, which provides execution infrastructure, and the AlgoResult class, which provides the metrics used for validation.
Sources: README.md:14-30, universal/algo.py:15-25, universal/result.py:27-35
Automated Unit Testing
The project utilizes pytest to automate the verification of algorithm properties. Testing focuses on specific behaviors like bias prevention, fee handling, and degenerate case performance.
Bias and Consistency Testing
A critical test ensures that algorithms do not have "forward bias." This is verified by running an algorithm on a full dataset and comparing its weights for earlier periods to a run performed only on that subset. If the weights match exactly (within an epsilon range), the algorithm is confirmed to be independent of future data points.
This diagram illustrates the forward bias check where weight history must remain consistent regardless of future data availability.
Sources: tests/test_algos.py:18-31
Edge Case Validation
Testing includes specific scenarios for individual algorithms:
- BAH (Buy and Hold): Validated to ensure that transaction fees do not affect total wealth, as the portfolio is never rebalanced.
- CRP (Constant Rebalanced Portfolio): Tested with degenerate weights (e.g.,
[1, 0, 0]) to ensure the portfolio wealth matches the underlying asset's wealth exactly. - TCO (Transaction Cost Optimization): Verified by setting extremely high fees to ensure the algorithm produces near-zero turnover.
Sources: tests/test_algos.py:34-60, tests/test_result.py:6-12
Data Simulation and Stress Testing
The project provides specialized tools in universal.tools to generate synthetic market data and perform statistical stress tests on algorithms.
Synthetic Portfolio Generation
The random_portfolio function generates asset prices assuming multivariate geometric Brownian motion. This allows developers to test algorithms against controlled drifts (mu), volatilities (sd), and correlations. It also supports injecting NaN values to test the robustness of algorithm implementations against missing data.
Sources: universal/tools.py:87-133
Bootstrap Testing
The bootstrap_algo strategy creates multiple simulated results by randomly dropping a fraction of trading days and reconstructing the time series. This provides a distribution of performance metrics rather than a single point estimate.
| Function | Purpose |
|---|---|
bootstrap_history | Reconstructs a time series by sampling a subset of returns without replacement. |
bootstrap_algo | Executes an algorithm across n bootstrapped samples in parallel using multiprocessing. |
random_portfolio | Generates a synthetic dataset for a given number of assets and time steps. |
Sources: universal/tools.py:615-633, universal/tools.py:87-90
Execution and Metrics Validation
Testing is not limited to correctness but also encompasses execution flow and metric accuracy.
The Execution Flow
Algorithms are typically tested using the quickrun utility, which executes the strategy, calculates timing, prints a summary, and plots the equity curve.
The sequence demonstrates the standard lifecycle of an algorithm test execution within the project.
Sources: universal/tools.py:75-84, universal/algo.py:85-115
Metrics and Result Verification
The AlgoResult class acts as the central hub for performance validation. It recalculates returns, logs returns, and handles the application of risk-free rates and fees. Testing involves verifying these calculated properties:
- Turnover: Validated via
to_rebalancelogic to ensure rebalancing math is correct. - Sharpe Ratio: Calculated with annualization based on detected data frequency.
- Drawdowns: Verified against the cumulative equity curve.
Sources: universal/result.py:112-145, universal/result.py:186-215, universal/result.py:302-315
Parameter and Subset Testing
To ensure robustness across different settings, the project supports grid testing and combinatorial analysis.
Combinatorial Analysis
The run_combination method allows testing an algorithm across a grid of parameters. This is used to verify that an algorithm performs consistently across its hyperparameter space.
Subset Testing
The run_subsets method executes an algorithm on all possible combinations of $r$ stocks from a larger dataset. This validates that the strategy is not overfitted to a specific set of assets.
Sources: universal/algo.py:171-190, universal/algo.py:228-261
Performance Profiling
For performance-critical algorithms, the project includes a profile tool using line_profiler. This strategy identifies bottlenecks in the step method, which is executed at every time point in the simulation.
Sources: universal/tools.py:58-72
Summary
Algorithm testing strategies in the universal-portfolios project bridge the gap between theoretical OLPS research and practical implementation. By combining strict unit tests for logic (bias and fees) with advanced statistical methods (bootstrapping and synthetic data), the project provides a framework for validating that portfolio weights are calculated correctly and that performance metrics accurately reflect the strategy's behavior in a simulated market environment.