Statistical Estimators & Covariance
Relevant source files
The following files were used as context for generating this wiki page:
Statistical estimators and covariance calculations form the mathematical backbone of many Online Portfolio Selection (OLPS) algorithms within this repository. They are primarily used to predict expected returns ($\mu$) and risk via covariance matrices ($\Sigma$), which are essential inputs for optimization-based strategies like Modern Portfolio Theory (MPT), Kelly Criterion, and Confidence Weighted Mean Reversion (CWMR).
The system is designed to be modular, allowing developers to plug in different estimation techniques—ranging from simple historical averages to complex shrinkage methods and external market assumptions—into standard portfolio algorithms.
Sources: universal/algos/mpt.py:17-30, universal/algos/estimators.py:91-100
Covariance Estimation Architecture
Covariance estimation is handled by specialized classes that wrap standard scikit-learn estimators or implement custom financial models. The primary wrapper, CovarianceEstimator, provides a unified interface for handling financial time-series data, including standardization, handling of zero-variance assets, and annualization of the resulting matrix.
Core Covariance Classes
| Class | Description |
|---|---|
CovarianceEstimator | A wrapper that accepts sklearn objects (e.g., LedoitWolf, OAS) and handles data preprocessing such as standardization and regularization. |
SingleIndexCovariance | Implements the Ledoit-Wolf shrinkage model combining sample covariance with a single-index model. |
ExponentiallyWeightedCovariance | Computes covariance where weights decay exponentially, giving more importance to recent data. |
JPMCovEstimator | Uses external market assumptions (J.P. Morgan assumptions) to construct a correlation-based covariance matrix. |
Sources: universal/algos/estimators.py:91-106, universal/algos/estimators.py:326-335, universal/algos/estimators.py:532-540, universal/algos/estimators.py:646-655
Estimation Data Flow
The following diagram illustrates how raw price data is processed into a covariance matrix suitable for portfolio optimization.
The flow ensures that data is clean and isotropic before entering the statistical model, then transformed back to meaningful financial scales.
Sources: universal/algos/estimators.py:109-158, universal/tools.py:378-385
Return ($\mu$) Estimation
Return estimators predict the future performance of assets. The project implements several approaches, from purely historical to Bayesian-like shrinkage models that incorporate "prior" beliefs about asset Sharpe ratios.
Sharpe-Based Estimation
The SharpeEstimator assumes that assets generally follow a global Sharpe ratio (defaulting to 0.4). It calculates expected return by multiplying the estimated volatility by this ratio and adjusting for the risk-free rate and asset-specific expenses.
Sources: universal/algos/estimators.py:161-193
Specialized Return Estimators
- HistoricalEstimator: Simply uses the historical mean of returns within a specified window, annualized by frequency.
- MixedEstimator: Combines historical returns with volatility-based returns using $\alpha$ (covariance regularization) and $\beta$ (weight on history) parameters.
- MLEstimator: Uses a
scikit-learnmodel to predict Sharpe ratios based on features like past Sharpe volatility and asset age. - JPMMeanEstimator: Maps assets to J.P. Morgan's Capital Market Assumptions to derive expected returns.
Sources: universal/algos/estimators.py:252-258, universal/algos/estimators.py:273-294, universal/algos/estimators.py:349-370, universal/algos/estimators.py:623-640
Integration with Portfolio Algorithms
Algorithms like MPT and Kelly utilize these estimators in their step method to determine the optimal weights $b$ for the next period.
Modern Portfolio Theory (MPT) Implementation
The MPT class can be configured with specific mu_estimator and cov_estimator strings or objects. In each step, it fits these models to the available history to solve objectives like maximizing Sharpe ratio or minimizing variance.
Kelly Criterion
The Kelly algo uses a rolling covariance matrix precomputed via tools.rolling_cov_pairwise. It applies a fraction of the Kelly weights, which are derived directly from the inverse of the covariance matrix multiplied by the expected excess return.
Sources: universal/algos/kelly.py:73-95, universal/tools.py:378-385
Utility and Helper Tools
The universal/tools.py file provides the underlying mathematical functions required by these estimators:
freq(ix): Calculates the number of data items per year based on a DatetimeIndex to handle annualization.opt_markowitz(mu, sigma, ...): A wrapper for the quadratic programming solver (cvxopt.solvers.qp) that finds optimal weights under constraints.rolling_cov_pairwise(df, ...): Efficiently computes rolling covariance across all asset pairs.
Sources: universal/tools.py:325-360, universal/tools.py:378-379, universal/tools.py:575-585
Summary of Configuration Parameters
| Parameter | Default | File / Line | Description |
|---|---|---|---|
global_sharpe | 0.4 | estimators.py:165 | Assumed annual Sharpe ratio for assets with unknown returns. |
w (Regularization) | 0.0 | estimators.py:101 | Shrinkage towards identity matrix in covariance (0 = sample, 1 = identity). |
alpha (Single Index) | None | single_index_covariance_estimator.py | Optimal shrinkage calculated automatically if None. |
reg (Kelly) | 0.0 | kelly.py:37 | Identity matrix regularization for singular covariance cases. |
Sources: universal/algos/estimators.py:101-165, universal/algos/kelly.py:37
Statistical estimation in this project prioritizes stability through regularization and flexibility through a wrapper architecture, ensuring that portfolio algorithms remain robust even with noisy or high-dimensional financial data.