Dependency & Binary Cache

Relevant source files

The following files were used as context for generating this wiki page:

• abx_dl/dependencies.py
• abx_dl/config.py
• abx_dl/models.py
• abx_dl/cli.py
• README.md
• TODOs.md

The Dependency & Binary Cache system in abx-dl manages the discovery, installation, and persistence of external tools (binaries) required by various plugins. It ensures that plugins have access to necessary executables like wget, chromium, or yt-dlp by coordinating between user-provided configurations and runtime-derived paths.

The system is designed to be lazy by default, auto-installing missing dependencies during the crawl phase unless configured otherwise. It utilizes a multi-layered configuration approach to separate static user preferences from dynamic, machine-specific binary locations, ensuring portability of user configurations across different environments.
Sources: README.md:92-108, abx_dl/config.py:1-12

Architecture and Data Layers

The dependency system operates across three distinct layers of data to manage binary resolution and state preservation:

1. User Configuration (config.env): Stores explicit user overrides (e.g., CHROME_BINARY=/usr/bin/google-chrome). These values are authoritative and if a path is provided, the system will not attempt to resolve it elsewhere.
2. Derived Cache (derived.env): Stores runtime-discovered information, such as absolute paths to installed binaries and the ABX_INSTALL_CACHE timestamp map. Entries here are cached for 24 hours.
3. Default Environment: System-level paths and default search locations provided by the abx-pkg providers.

Sources: abx_dl/config.py:10-18, README.md:144-160

Configuration Separation Logic

The system strictly separates these layers to ensure that config.env remains portable while derived.env contains machine-specific paths.

The diagram shows the priority logic used to resolve a binary path, starting from user overrides down to dynamic resolution.
Sources: abx_dl/config.py:125-145, README.md:155-165

Binary Resolution Process

Binary resolution is driven by the interaction between abx_dl/dependencies.py and the abx-pkg library. The system supports multiple package managers through specific providers.

Supported Providers

The following providers are initialized in order of priority:

• EnvProvider: Searches the system $PATH.
• PipProvider: Manages binaries in ~/.config/abx/lib/{arch}/pip/venv/.
• NpmProvider: Manages binaries in ~/.config/abx/lib/{arch}/npm/.
• BrewProvider: Interfaces with Homebrew.
• AptProvider: Interfaces with the system advanced package tool.

Sources: abx_dl/dependencies.py:14-38, README.md:183-190

Binary Request Lifecycle

The lifecycle transitions from a requirement declaration in a plugin's config.json to a resolved Binary record.

Event / Record	Description	Metadata Included
RequiredBinary	The requirement defined in plugin config.json.	Name, min_version, providers.
BinaryRequestEvent	Emitted to trigger providers to find/install the binary.	Name, plugin_name, timeout.
Binary record	The final resolved object emitted by a provider.	abspath, version, sha256.

Sources: abx_dl/models.py:75-84, TODOs.md:17-25

Installation Phase

The installation phase is a distinct pre-run lifecycle event where abx-dl ensures all dependencies for enabled plugins are met.

This sequence diagram illustrates the flow of events during the abx-dl install command or the auto-install trigger.
Sources: README.md:111-125, TODOs.md:27-35

Key Functions

• load_binary(spec): Attempts to locate a binary without installing it.
• install_binary(spec): Attempts to locate a binary, and if missing, invokes the load_or_install() method of the abx-pkg Binary object.
• get_required_binary_requests(plugin, binaries, ...): Hydrates raw binary requirements into actionable request payloads by expanding variables in the binary name.

Sources: abx_dl/dependencies.py:41-79, abx_dl/config.py:440-461

Directory Structure

The system installs binaries and stores cache files in architecture-specific subdirectories under the user's home directory.

Path	Purpose
~/.config/abx/config.env	User-defined binary overrides and settings.
~/.config/abx/derived.env	Cached binary paths and install timestamps.
~/.config/abx/lib/{arch}/bin	General binary installation directory.
~/.config/abx/lib/{arch}/pip	Python virtual environments for dependencies.
~/.config/abx/lib/{arch}/npm	Node.js modules and binary symlinks.

Sources: abx_dl/config.py:22-38, abx_dl/config.py:330-350

Implementation Details

Binary Record Representation

In the CLI, binary results are represented by the _BinaryRecord class, which prioritizes displaying the absolute path and version string.

class _BinaryRecord:
    def __init__(self, name, abspath, plugin, hook_name, status, version="", error="", order=0):
        self.name = name
        self.abspath = abspath
        self.version = version
        # ...
    @property
    def display_output(self) -> str:
        if self.error: return self.error
        parts = [self.abspath, self.version] if self.abspath else [self.name]
        return " ".join(filter(None, parts))

Sources: abx_dl/cli.py:537-565

Path Normalization for Environment

When preparing the environment for a plugin hook, abx-dl prepends the runtime-managed bin directories to the PATH variable. This ensures that the versions installed in LIB_DIR are preferred over system-wide binaries unless explicitly overridden.
Sources: abx_dl/models.py:205-220

Cache Validation

Successful installs are cached for 24 hours in derived.env. At runtime, abx-dl validates cached absolute paths. If a cached path is found to be stale (e.g., the file was deleted), the system ignores the cache entry and falls back to the normal provider resolution flow.
Sources: README.md:155-165