Adding a New Import Source

Relevant source files

The following files were used as context for generating this wiki page:

• sofl/importer/source.py
• sofl/importer/importer.py
• sofl/importer/location.py
• sofl/importer/steam_source.py
• sofl/importer/lutris_source.py
• sofl/importer/bottles_source.py
• sofl/main.py
• sofl/preferences/import_sources.py

The game import system in SOFL is designed to be modular, allowing developers to easily add support for new game launchers and sources. By creating a new "Import Source," you can teach SOFL how to find games from other platforms, parse their metadata, and add them to the central library. This system is crucial for aggregating games from various launchers like Steam, Lutris, Heroic, and more into a single, unified interface.

This guide outlines the architecture of the import system and provides a step-by-step process for creating and integrating a new import source. The core components are the Source class, which defines the source's properties, the SourceIterable class, which contains the logic for finding games, and the Location class, which helps locate necessary files and directories.

Architecture Overview

The import process is orchestrated by the Importer class, which manages multiple Source objects. Each Source is responsible for a specific platform (e.g., Steam) and uses a corresponding SourceIterable to discover games.

Core Components

Component	Description	Relevant File
Importer	Orchestrates the entire import process. It runs each enabled source in a separate thread and manages progress reporting and error handling.	sofl/importer/importer.py
Source	An abstract base class representing a source of games (e.g., a launcher). It defines metadata like source_id and name.	sofl/importer/source.py
SourceIterable	A companion class to Source. Its __iter__ method contains the logic to find and yield Game objects from the source's data files or databases.	sofl/importer/source.py
Location	A helper class used within a Source to find required directories or files, searching through a list of candidates paths.	sofl/importer/location.py
Game	The data model representing a single game in the SOFL library.	sofl/game.py

Class Relationships

The following diagram illustrates the relationships between the main classes in the import system. A new implementation will typically involve creating subclasses of Source and SourceIterable.

Import Process Flow

The import process is initiated by the user or automatically on startup. The Importer class manages the entire workflow asynchronously.

1. The SOFLApplication creates an Importer instance.
2. Enabled Source objects (e.g., SteamSource, LutrisSource) are added to the Importer.
3. Importer.run() is called, which creates a Gio.Task for each source.
4. Each task runs source_task_thread_func, which iterates over the Source object.
5. Iterating the Source calls its __iter__ method, which in turn calls the __iter__ method of its SourceIterable.
6. The SourceIterable yields Game objects, which are processed and added to the Store.

This sequence diagram shows the high-level interaction during an import.

Step 1: Create the Source Class

The first step is to define a new class that inherits from sofl.importer.source.Source or one of its subclasses, like URLExecutableSource. This class acts as a blueprint for your import source.

Most sources use a custom URL scheme (e.g., steam://rungameid/{game_id}) to launch games. For these, inheriting from URLExecutableSource is recommended as it provides a platform-agnostic make_executable method.

Source Class Properties

Property	Type	Description	Required
source_id	str	A unique, lowercase identifier for the source (e.g., "steam", "lutris").	Yes
name	str	The human-readable name displayed in the UI (e.g., "Steam").	Yes
iterable_class	type[SourceIterable]	A reference to the companion SourceIterable class.	Yes
available_on	set[str]	A set of platforms ("linux", "win32", "darwin") where the source is available.	Yes
locations	NamedTuple	A NamedTuple containing Location objects for required files/directories.	Yes
url_format	str	The URL template for launching games. Required if inheriting from URLExecutableSource.	Yes
game_id_format	str	A format string to create unique game IDs. Defaults to {source_id}_{game_id}.	No

Example: SteamSource

SteamSource inherits from URLExecutableSource and defines the necessary properties to find and launch Steam games.

# sofl/importer/steam_source.py:89-122
class SteamLocations(NamedTuple):
    data: Location

class SteamSource(URLExecutableSource):
    source_id = "steam"
    name = _("Steam")
    available_on = {"linux", "win32", "darwin"}
    iterable_class = SteamSourceIterable
    url_format = "steam://rungameid/{game_id}"

    locations: SteamLocations

    def __init__(self) -> None:
        super().__init__()
        self.locations = SteamLocations(
            Location(
                schema_key="steam-location",
                candidates=(
                    shared.home / ".steam" / "steam",
                    shared.data_dir / "Steam",
                    # ... other candidate paths
                ),
                paths={
                    "libraryfolders.vdf": LocationSubPath(
                        "steamapps/libraryfolders.vdf"
                    ),
                    "librarycache": LocationSubPath("appcache/librarycache", True),
                },
                invalid_subtitle=Location.DATA_INVALID_SUBTITLE,
            )
        )

Sources: sofl/importer/steam_source.py:89-122

Step 2: Implement the SourceIterable

The SourceIterable class contains the core logic for discovering games. Its __iter__ method must be a generator that yields Game objects.

__iter__ Method Logic

The generator should:

1. Access the resolved file paths from self.source.locations.
2. Open and parse the relevant data source (e.g., SQLite database, YAML/JSON/VDF file).
3. Loop through the entries representing installed games.
4. For each game, create a Game object using GameFactory.create_game().
5. Yield the Game object, optionally with a dictionary of additional_data like a path to a local cover image.
6. Handle any exceptions during iteration to prevent the entire import from failing.

The following flowchart shows the typical logic inside the __iter__ method.

Example: BottlesSourceIterable

This iterable reads a library.yml file, parses it, and yields games found within. It also constructs a path to a local thumbnail.

# sofl/importer/bottles_source.py:27-65
class BottlesSourceIterable(SourceIterable):
    source: "BottlesSource"

    def __iter__(self):
        """Generator method producing games"""

        data = self.source.locations.data["library.yml"].read_text("utf-8")
        library: dict = yaml.safe_load(data)

        for entry in library.values():
            # Build game
            values = {
                "name": entry["name"],
                "source": self.source.source_id,
                "game_id": self.source.game_id_format.format(game_id=entry["id"]),
                "executable": self.source.make_executable(
                    bottle_name=entry["bottle"]["name"],
                    game_name=entry["name"],
                ),
                "added": shared.import_time,
            }
            game = GameFactory.create_game(values)

            # ... logic to find cover path ...

            additional_data = {}
            if entry["thumbnail"]:
                image_name = entry["thumbnail"].split(":")[1]
                image_path = bottles_location / bottle_path / "grids" / image_name
                additional_data = {"local_image_path": image_path}

            yield (game, additional_data)

Sources: sofl/importer/bottles_source.py:27-65

Step 3: Define Locations

The Location class is a robust helper for finding the necessary files and directories for your source. In your Source's __init__ method, you define one or more Location objects.

Each Location is configured with:

• schema_key: The GSettings key where a user-overridden path is stored.
• candidates: A tuple of potential default paths to check. The first valid path found is used.
• paths: A dictionary mapping a key to a LocationSubPath object, which represents a required file or directory relative to the location's root.
• invalid_subtitle: A message to show the user if they select an invalid directory in the preferences.

When location.resolve() is called, it tries to find a valid root path from user settings or the candidates. If successful, you can access the resolved sub-paths like location['my_file']. If it fails for a non-optional path, it raises UnresolvableLocationError.

Sources: sofl/importer/location.py, sofl/importer/steam_source.py:106-122

Step 4: Integrate into the Application

After creating the Source and SourceIterable classes, you need to integrate them into the application's UI and import logic.

1. Add to Preferences

To allow users to enable/disable your source and set a custom path, you must modify the preferences dialog.

1. Add UI Widgets: Add an AdwExpanderRow and other necessary widgets to data/gtk/preferences.ui. The object IDs should follow the convention {source_id}_expander_row, {source_id}_{location_name}_action_row, etc.
2. Initialize the Row: In sofl/preferences/import_sources.py, add your new source class to the list of source_classes to be initialized. The ImportSourcesSection.init_source_row method will automatically handle binding the GSettings switch and connecting the "Choose Folder" button.

# sofl/preferences/import_sources.py:29-44
class ImportSourcesSection:
    def __init__(self, dialog: "SOFLPreferences") -> None:
        # ... other sources
        from sofl.importer.steam_source import SteamSource
        # Add your new source here
        # from sofl.importer.my_new_source import MyNewSource

        for source_class in (
            BottlesSource,
            # ...
            SteamSource,
            # MyNewSource,
        ):
            source = source_class()
            if not source.is_available:
                # ... hide row
            else:
                self.init_source_row(source)

Sources: sofl/preferences/import_sources.py:29-44, 70-87

2. Add to Main Import Logic

Finally, add your source to the main import action in sofl/main.py. This ensures that when an import is triggered, your source is included if it's enabled in the settings.

# sofl/main.py:302-305
    def on_import_action(self, *_args: Any) -> None:
        shared.importer = Importer()

        if shared.schema.get_boolean("steam"):
            shared.importer.add_source(SteamSource())
        
        # Add your new source here, guarded by its GSettings key
        # if shared.schema.get_boolean("my_new_source"):
        #     shared.importer.add_source(MyNewSource())

        # ... other sources

        shared.importer.run()

Sources: sofl/main.py:302-328

Summary

Adding a new import source is a structured process that leverages SOFL's modular architecture. By creating Source and SourceIterable subclasses, you encapsulate the platform-specific logic for game discovery. The Location class simplifies the task of finding necessary files, and integration into the UI and main application logic is straightforward. This design allows SOFL to be easily extended to support a wide array of game launchers on Linux.