DataFrame & Group Operations

Relevant source files

The following files were used as context for generating this wiki page:

• alphapy/frame.py
• alphapy/group.py
• alphapy/analysis.py
• alphapy/sport_flow.py
• alphapy/market_flow.py
• alphapy/system.py

DataFrame and Group operations form the core data management layer of AlphaPy. This system provides a structured way to handle collections of related data (Groups) and their underlying tabular representations (Frames). It facilitates the ingestion, transformation, and storage of datasets used in both MarketFlow and SportFlow pipelines.

The framework leverages pandas for in-memory data manipulation while providing abstraction layers to manage namespaces through the Space class. This allows for organized handling of multiple data fractals (e.g., 1d, 5m), subjects (e.g., stock, game), and schemas (e.g., prices, scores).

Sources: alphapy/frame.py:34-40, alphapy/group.py:42-50, alphapy/market_flow.py:245-255

Group Management

The Group class is a container for common members, such as stock tickers or sports teams. It allows for the definition of dynamic or fixed collections that share a common Space.

Group Architecture

Groups are stored in a class-level dictionary Group.groups to ensure uniqueness across the application. They support membership operations including adding and removing identifiers, provided the group is marked as dynamic.

Key Group Operations

Operation	Description
add(newlist)	Adds a list of string identifiers to the group if dynamic is True.
remove(remlist)	Removes specified members from the group if dynamic is True.
member(item)	Checks if a specific item exists within the group set.

Sources: alphapy/group.py:100-165

Frame Infrastructure

The Frame class manages pandas.DataFrame objects in memory, associating them with a specific name and Space. This ensures that data remains context-aware throughout the pipeline.

Frame Naming Convention

Names are generated using the frame_name function, which joins the group name with space attributes (subject, schema, fractal) using underscores. For example, a "tech" group in a 1d stock price space results in tech_stock_prices_1d.

Sources: alphapy/frame.py:43-66

Data Flow for Frame Loading

The load_frames function handles the retrieval of data from the filesystem. It can either load a single consolidated file for a group or split files for individual members.

Data Transformations and Sequencing

AlphaPy provides specialized operations to prepare DataFrames for machine learning, primarily focusing on time-series sequencing and feature delta generation.

Feature Sequencing

The sequence_frame function transforms a standard DataFrame into a supervised learning format by creating lagging and leading value sequences. This includes:

• Laggards: Features shifted by a defined lag_period.
• Leaders: Features contemporaneous with the target that are preserved without shifting.
• Target: The prediction variable, shifted by the forecast_period.

Sources: alphapy/frame.py:270-316, alphapy/analysis.py:126-140

Technical Implementation of Sequencing

# Preserve leader columns
new_cols.append(df[le_cols])
new_names += [le_cols[j] for j in range(le_len)]

# Forecast Target(s)
new_cols.append(pd.DataFrame(df[target].shift(1-forecast_period)))
new_names.append(target)

Sources: alphapy/frame.py:305-312

Sport-Specific Group Operations

In SportFlow, the system performs complex merges between team-level statistics and game-level frames.

Team Statistic Generation

The generate_team_frame function calculates rolling statistics (wins, losses, streaks, margins) for a specific team by iterating through a seasonal game frame.

Model Frame Merging

The insert_model_data function populates a master model frame by looking up team statistics based on Date and Team identifiers.

Parameter	Type	Description
mf	DataFrame	The destination master model frame.
tf	DataFrame	The source team-specific frame.
mdict	dict	Dictionary of fields to insert.
prefix	str	Prefix (e.g., 'home' or 'away') for the target columns.

Sources: alphapy/sport_flow.py:417-446

Persistence Operations

Frames are saved and loaded using standardized I/O functions that support custom delimiters and extensions.

• read_frame: Loads a CSV/delimited file into a pandas.DataFrame using low_memory=False and optional index column specification.
• write_frame: Exports a DataFrame to the filesystem, commonly used to save processed input files for the AlphaPy pipeline.
• dump_frames: A bulk operation that saves every member of a Group to separate files.

Sources: alphapy/frame.py:105-136, alphapy/frame.py:143-174, alphapy/frame.py:233-264

Summary

DataFrame and Group operations in AlphaPy bridge the gap between raw data storage and machine learning model requirements. By utilizing a class-based tracking system (Frame.frames and Group.groups) and specialized sequencing logic, the framework ensures that time-series data is correctly partitioned and formatted for predictive analysis across different domains like financial markets and sports analytics.

Sources: alphapy/analysis.py:114-124, alphapy/market_flow.py:265-275