Metadata Fetching Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• sofl/store/pipeline.py
• sofl/store/managers/sgdb_manager.py
• sofl/store/managers/steam_api_manager.py
• sofl/importer/importer.py
• sofl/importer/steam_source.py
• sofl/utils/steam.py
• sofl/main.py

The Metadata Fetching Pipeline is an asynchronous system responsible for enriching Game objects with detailed information and artwork. When a game is discovered by the Importer, a new pipeline is created for it. This pipeline processes the game through a series of "managers," each tasked with fetching a specific piece of data from local files or remote APIs, such as the Steam API and SteamGridDB.

This modular, asynchronous approach ensures that the user interface remains responsive while game data is populated in the background. The pipeline manages the state and progress of each game's enrichment process, from initial discovery to the final display in the user's library.

Sources: sofl/store/pipeline.py, sofl/importer/importer.py

Architecture Overview

The metadata fetching process involves three core components: the Importer, the Pipeline, and the Managers.

1. Importer: Scans various sources (e.g., Steam, Lutris) for games. When a new game is found, it creates a basic Game object.
2. Store/Pipeline Creation: The Importer passes the new Game object to the central Store, which initiates a Pipeline for it.
3. Pipeline Execution: The Pipeline runs the Game object through a series of registered Managers.
4. Managers: Each Manager is a specialized module that fetches specific data (e.g., developer info, cover art) and updates the Game object.

The following diagram illustrates the high-level interaction between these components.

The Pipeline Class

The Pipeline class is the core of the metadata fetching process. An instance of this class is created for each new game that is imported. It manages the execution of various data-fetching tasks and tracks the overall progress.

Sources: sofl/store/pipeline.py

Key Properties and Methods

Property/Method	Description
game	The Game object being processed by the pipeline.
managers	A list of Manager instances that will be run for the game.
progress	A float value (0.0 to 1.0) representing the completion percentage of the pipeline.
is_done	A boolean that is True when all managers have finished processing.
run()	Starts the pipeline, iterating through its managers and executing them.
advance()	A callback method executed by a manager to signal progress. It updates the pipeline's progress and emits the advanced signal.
advanced (signal)	A GObject signal emitted whenever a manager completes its task, allowing other parts of the application (like the Importer) to monitor progress.

Sources: sofl/store/pipeline.py:21-78

Class Structure

The diagram below shows the structure of the Pipeline class.

Pipeline Execution Flow

The pipeline execution is initiated by the Importer after a new game is discovered.

1. The Importer's thread function (source_task_thread_func) finds a game.
2. It calls shared.store.add_game(), which creates and returns a Pipeline instance.
3. The Importer connects to the pipeline's advanced signal to monitor its progress.
4. The pipeline's run() method is called, which iterates through all enabled managers.
5. Each manager performs its specific task (e.g., an API call) and then calls the pipeline's advance() method.
6. advance() updates the progress and emits the advanced signal.
7. The Importer receives this signal and updates its overall progress tracking.

The following sequence diagram illustrates this flow:

Managers

Managers are modular components responsible for specific data-fetching and processing tasks. They are registered in the main application and passed to the Pipeline upon its creation.

Sources: sofl/main.py:133-137

Key Managers

Manager	Responsibility	Source File
FileManager	Loads and saves game data to/from JSON files on disk.	sofl/store/managers/file_manager.py
DisplayManager	Manages how the game is displayed in the UI.	sofl/store/managers/display_manager.py
CoverManager	Handles local cover images and downloads missing ones.	sofl/store/managers/cover_manager.py
SteamAPIManager	Fetches game metadata, such as the developer, from the Steam Web API.	sofl/store/managers/steam_api_manager.py
SgdbManager	Searches for and downloads game artwork (grids/covers) from SteamGridDB.	sofl/store/managers/sgdb_manager.py

SteamAPIManager

This manager enriches the Game object with data from the Steam Web API. Its primary responsibility is to fetch the developer's name.

• Operation: It uses the SteamAPIHelper utility to make a request to the https://store.steampowered.com/api/appdetails endpoint.
• Rate Limiting: The SteamAPIHelper includes a SteamRateLimiter to avoid exceeding the API's request limit (200 requests per 5 minutes). The rate limiter's state is persisted between application runs.
• Data Structure: The fetched data is parsed into a SteamAPIData typed dictionary.

Sources: sofl/store/managers/steam_api_manager.py:31-43, sofl/utils/steam.py:65-131

The SteamAPIData structure is defined as follows:

class SteamAPIData(TypedDict):
    """Dict returned by SteamAPIHelper.get_api_data"""

    developer: str

Sources: sofl/utils/steam.py:43-47

SgdbManager

This manager is responsible for finding and downloading game artwork from SteamGridDB.

• Operation:
  1. It first checks if the game already has a cover.
  2. If not, it calls search_game() to find the game's ID on SteamGridDB.
  3. With the ID, it calls get_grids() to fetch a list of available grid images.
  4. It downloads the first suitable image and saves it, updating the Game object with the path.
• Rate Limiting: Like the Steam manager, it uses a rate limiter to manage API calls to SteamGridDB.

Sources: sofl/store/managers/sgdb_manager.py:42-83

Integration with the Importer

The Importer class is the entry point for the entire metadata fetching process. It is responsible for discovering games and initiating a pipeline for each one.

When an import is triggered, the Importer iterates through enabled sources (like SteamSource). The source yields a basic Game object. The Importer then calls shared.store.add_game(game), which handles the creation of the Pipeline. The Importer collects all active pipelines in a set (self.game_pipelines) and monitors their collective progress to update the UI, such as the import progress bar.

# sofl/importer/importer.py

# ... inside source_task_thread_func ...
pipeline: Pipeline = shared.store.add_game(game, additional_data)
if pipeline is not None:
    logging.info("Imported %s (%s)", game.name, game.game_id)
    pipeline.connect(
        "advanced",
        self.pipeline_advanced_callback,
    )
    self.game_pipelines.add(pipeline)

Sources: sofl/importer/importer.py:192-200

This design decouples the game discovery logic (in Importer and Sources) from the data enrichment logic (in Pipeline and Managers), creating a clean and maintainable architecture.

Sources: sofl/importer/importer.py:31-60