External API Integrations
Relevant source files
The following files were used as context for generating this wiki page:The SOFL application enhances the user's game library by integrating with several external web services. These integrations range from deep API-level data fetching for game metadata and artwork to simpler URL-based searches that open in the user's web browser. The primary goals are to automate the retrieval of game information, such as developers and cover art, providing a richer and more visually appealing library.
Key integrated services include the Steam Storefront API for game details and SteamGridDB for game artwork. The system is designed with robustness in mind, featuring a sophisticated rate-limiting mechanism to prevent API abuse and ensure compliance with service terms. Other services like IGDB, ProtonDB, and Lutris are integrated via simple web search links.
Core Architecture
The external integrations are primarily managed through a system of "Managers" and "Helpers". Managers, such as SteamAPIManager and SgdbManager, are part of the application's data processing pipeline. They orchestrate the data fetching process for games. Helpers, like SteamAPIHelper and SgdbHelper, encapsulate the logic for making direct HTTP requests to the external APIs. A crucial component of this architecture is the RateLimiter, which ensures that all API requests adhere to the respective service's limits.
Sources: sofl/main.py:53-56, sofl/utils/steam.py:100-103, sofl/utils/steamgriddb.py:104-106
Data Fetching Flow
The following diagram illustrates the general sequence for fetching external data when a new game is imported.
This diagram shows how a manager uses a helper to fetch data, with the helper first acquiring a token from the rate limiter before making the external API call.
Sources: sofl/store/managers/steam_api_manager.py:30-46, sofl/utils/steam.py:108-115
Steam Integration
Integration with Steam is multifaceted, involving both parsing local Steam files and querying the official Steam Storefront API.
Sources: sofl/utils/steam.py, sofl/importer/steam_source.py
Steam File Parsing
Before any API calls are made, SOFL parses local Steam files to identify installed games. The SteamSource class iterates through appmanifest_*.acf files located in Steam library folders. The SteamFileHelper class is responsible for parsing these manifest files to extract essential data.
This flow shows how local Steam manifest files are used to gather initial game information.
Sources: sofl/importer/steam_source.py:38-53, sofl/utils/steam.py:82-97
The get_manifest_data function uses regular expressions to extract key-value pairs from the manifest file.
| Key | Regex | Description |
|---|---|---|
name | "name"\s+"(.*)" | The name of the game. |
appid | "appid"\s+"(.*)" | The unique Steam Application ID. |
stateflags | "stateflags"\s+"(.*)" | Flags indicating the game's installation status. |
Sources: sofl/utils/steam.py:91-95
Steam Storefront API
Once a game's appid is known, the SteamAPIHelper class is used to fetch additional details from the Steam Storefront API.
Key Components
SteamAPIHelper: A class that provides theget_api_datamethod to query the API.SteamAPIManager: A manager that integratesSteamAPIHelperinto the game processing pipeline, specifically to fetch the developer's name.- Endpoint:
https://store.steampowered.com/api/appdetails?appids={appid}
Sources: sofl/utils/steam.py:101-103, sofl/store/managers/steam_api_manager.py:18-20
The get_api_data method returns a SteamAPIData dictionary containing the developer's name. It handles cases where the app is not found or is not a game (e.g., DLC, software).
# sofl/utils/steam.py:105-129
def get_api_data(self, appid: str) -> SteamAPIData:
"""
Get online data for a game from its appid.
May block to satisfy the Steam web API limitations.
See https://wiki.teamfortress.com/wiki/User:RJackson/StorefrontAPI#appdetails
"""
# Get data from the API (way block to satisfy its limits)
with self.rate_limiter:
try:
with requests.get(
f"{self.base_url}/appdetails?appids={appid}", timeout=5
) as response:
response.raise_for_status()
data = response.json()[appid]
except HTTPError as error:
logging.warning("Steam API HTTP error for %s", appid, exc_info=error)
raise error
# Handle not found
if not data["success"]:
logging.debug("Appid %s not found", appid)
raise SteamGameNotFoundError()
# Handle appid is not a game
if data["data"]["type"] not in {"game", "demo", "mod"}:
logging.debug("Appid %s is not a game", appid)
raise SteamNotAGameError()
# Return API values we're interested in
values = SteamAPIData(developer=", ".join(data["data"]["developers"]))
return values
SteamGridDB Integration
SOFL uses SteamGridDB to automatically download game cover art. This integration requires an API key, which the user can configure in the application's preferences.
Sources: sofl/utils/steamgriddb.py, sofl/preferences.py:165-175
Configuration
The SteamGridDB API key is managed in the preferences window.
- UI Element:
sgdb_key_entry_rowinsofl/preferences.py. - GSettings Key:
sgdb-key. - The UI provides a link to the SteamGridDB API key generation page.
Sources: sofl/preferences.py:165-175
API Interaction
The SgdbHelper class handles all communication with the SteamGridDB API. It uses the provided API key in the Authorization header for all requests.
The following endpoints are used:
| Method | Endpoint | Description |
|---|---|---|
GET | /grids/game/{game_id} | Fetches grids (cover art) for a given Steam App ID. |
GET | /grids/search/autocomplete/{term} | Searches for games by name to find a matching game ID. |
Sources: sofl/utils/steamgriddb.py:112-140
The process to fetch a cover is as follows:
- The
SgdbManageris triggered to find a cover for a game. - If the game has a Steam App ID,
SgdbHelper.get_grids_by_gameidis called. - If not,
SgdbHelper.search_for_gameis called to find the game's ID on SteamGridDB, then step 2 is executed. - The helper downloads the image URL corresponding to the desired grid.
Sources: sofl/utils/steamgriddb.py:112-140, sofl/main.py:55-56
Generic Search Integrations
For several other services, SOFL provides a simpler integration by constructing a search URL and opening it in the user's default web browser. These actions are triggered from the game's context menu.
The SOFLApplication class contains methods like on_igdb_search_action that call a generic search method.
# sofl/main.py:423-424
def search(self, uri: str) -> None:
Gio.AppInfo.launch_default_for_uri(f"{uri}{quote(shared.win.active_game.name)}")
The following table summarizes the configured search providers:
| Service | Method in main.py | Search URL Format |
|---|---|---|
| IGDB | on_igdb_search_action | https://www.igdb.com/search?type=1&q= |
| SteamGridDB | on_sgdb_search_action | https://www.steamgriddb.com/search/grids?term= |
| ProtonDB | on_protondb_search_action | https://www.protondb.com/search?q= |
| Lutris | on_lutris_search_action | https://lutris.net/games?q= |
| HowLongToBeat | on_hltb_search_action | https://howlongtobeat.com/?q= |
Sources: sofl/main.py:423-438
Rate Limiting System
To avoid being blocked by external services, SOFL implements a token bucket-based rate-limiting system. A generic RateLimiter class is defined and then subclassed for specific services.
Sources: sofl/utils/rate_limiter.py
This diagram shows the inheritance structure of the rate limiters.
Sources: sofl/utils/rate_limiter.py:18, sofl/utils/steam.py:61, sofl/utils/steamgriddb.py:84
Rate Limiter Configurations
| Service | Class | Refill Period (s) | Refill Tokens | Burst Tokens |
|---|---|---|---|---|
| Steam | SteamRateLimiter | 300 (5 minutes) | 200 | 100 |
| SteamGridDB | SgdbRateLimiter | 60 (1 minute) | 60 | 30 |
Sources: sofl/utils/steam.py:65-67, sofl/utils/steamgriddb.py:88-90
State Persistence
The SteamRateLimiter is unique in that it persists its history of API requests across application sessions. It uses GSettings to store the timestamps of recent requests in the steam-limiter-tokens-history key. This ensures that the rate limit (200 requests per 5 minutes) is respected even if the application is restarted frequently.
Sources: sofl/utils/steam.py:72-76, sofl/utils/steam.py:80-81
Summary
SOFL's external API integrations are a critical feature for automating the enrichment of the game library. The system uses a combination of deep API integrations with services like Steam and SteamGridDB, and simpler web-based searches for others. The architecture is robust, featuring a manager/helper pattern and a sophisticated, persistent rate-limiting system to ensure stability and compliance with external service policies. This functionality significantly improves the user experience by providing detailed, accurate, and visually appealing game information with minimal manual effort.