MarketFlow Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The MarketFlow Pipeline is a specialized machine learning framework within the AlphaPy project designed for financial market analysis and speculation. Its primary purpose is to automate the end-to-end process of acquiring market data, engineering technical features, training predictive models, and executing trading systems based on those predictions.
The pipeline integrates various data sources (such as Yahoo Finance, IEX, and Quandl) and utilizes scikit-learn and other advanced estimators (XGBoost, LightGBM) to forecast market movements. It operates in two primary modes: a model-creation mode for training and an analysis mode for running trading systems and portfolio simulations.
Sources: README.rst:44-55, alphapy/market_flow.py:166-177
Pipeline Architecture and Flow
The market_pipeline function serves as the central orchestration point. It processes a Model object and a set of market_specs to execute a multi-stage workflow.
Execution Workflow
- Group Definition: Identifies the target group of market symbols to be analyzed.
- Data Acquisition: Retrieves historical or intraday data from external APIs or local storage.
- Feature Application: Applies technical indicators and functions to the raw price data.
- Analysis Execution: Creates an
Analysisobject to handle data lagging, forecasting, and splitting for training/testing. - System Execution: If configured, runs a
System(trading logic) and generates a portfolio.
The following flowchart illustrates the high-level execution path of the MarketFlow pipeline:
Data Acquisition and Management
The pipeline uses get_market_data to fetch data based on the data_fractal (e.g., daily, 5-minute) and schema defined in the configuration.
Data Dispatching
The system utilizes a dispatch table to route requests to various financial data providers. Supported providers include:
- Google: Historically supported, now marked as unavailable.
- IEX: Fetches daily and historical intraday data.
- Pandas Web Reader: General purpose data retrieval.
- Quandl: Handles specialized sub-feeds.
- Yahoo: Custom implementation for intraday and daily data.
Sources: alphapy/data.py:598-602, alphapy/data.py:657-670
Configuration Parameters
The market.yml configuration file controls the pipeline's behavior. Key parameters include:
| Parameter | Type | Description |
|---|---|---|
create_model | Boolean | Whether to train a new machine learning model. |
data_fractal | String | The timeframe of the source data (e.g., '1d', '5m'). |
forecast_period | Integer | The number of periods ahead to predict. |
lag_period | Integer | The number of historical periods used as features. |
schema | String | The data source/provider (e.g., 'yahoo', 'iex'). |
target_group | String | The name of the symbol group defined in config. |
Sources: alphapy/market_flow.py:73-110
Analysis and Model Training
If create_model is enabled, the pipeline executes the Analysis phase. This involves sequence_frame, which aligns the target variable with historical features based on the forecast_period.
Feature Engineering
Features are applied globally to symbol groups using vmapply. This includes applying functions defined in the market.yml to specific columns. The pipeline also initializes default probability variables:
phigh: probability >= 0.7plow: probability <= 0.3
Sources: alphapy/market_flow.py:133-136, alphapy/market_flow.py:219-221
Trading Systems and Portfolio Generation
A unique aspect of MarketFlow is the ability to run a System immediately after model training or prediction. A system is defined by entry and exit rules:
- Long Entry/Exit: Logic for buying and selling.
- Short Entry/Exit: Logic for shorting and covering.
- Hold Period: Time-based exit constraint.
- Scale: Boolean indicating if position scaling is used.
Once the system runs via run_system, it outputs Trade Frame series (tfs), which are then consumed by gen_portfolio to produce performance metrics.
Sources: alphapy/market_flow.py:229-245, alphapy/globals.py:180-205
Directory Structure
The pipeline automatically manages a standardized directory structure for every project:
config/: YAML configuration files.data/: Raw market data.input/: Processed data ready for the AlphaPy main pipeline.model/: Saved predictors (.pklor.h5) and feature maps.output/: Predictions, probabilities, and rankings.plots/: Visualizations (ROC curves, confusion matrices).systems/: Trading system outputs and portfolio results.
Sources: alphapy/market_flow.py:307-313, alphapy/main.py:456-461
Conclusion
The MarketFlow Pipeline provides a robust infrastructure for quantitative finance, bridging the gap between raw data retrieval and actionable trading systems. By combining configurable data providers with the full AlphaPy machine learning stack, it allows for rapid prototyping and validation of market-based predictive models.