Home

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• abx_dl/cli.py
• abx_dl/models.py
• server/server.py
• pyproject.toml
• skills/abx-dl/SKILL.md
• TODOs.md

abx-dl is an all-in-one Command Line Interface (CLI) tool and web server designed to auto-detect and download content from URLs using the ArchiveBox plugin ecosystem. It provides a simplified interface for scraping, downloading media, and digital preservation without requiring a full ArchiveBox installation. The tool leverages a collection of specialized plugins—such as wget, yt-dlp, and chrome—to extract metadata, media, and rendered snapshots of web pages.

Sources: README.md:1-15, skills/abx-dl/SKILL.md:1-10

System Architecture

The architecture of abx-dl is built upon the ABX Plugin Library, which manages the lifecycle of downloading and scraping tools through distinct phases and an event-driven bus.

Core Components

The system consists of several integrated modules:

• CLI Interface: A rich terminal interface powered by rich-click that handles user commands and provides live progress feedback.
• Web Server: A Flask-based server providing a REST API and a Web UI for managing download sessions.
• Orchestrator & Event Bus: Manages hook execution, event propagation, and state transitions between different phases of a download.
• Plugin System: Discovers and loads plugins from the abx-plugins package, executing logic via language-agnostic executable "hooks".

Sources: README.md:330-340, abx_dl/cli.py:40-60, server/server.py:80-100

Execution Workflow

The system operates in a sequential pipeline to ensure dependencies are met before extraction begins.

The workflow ensures that BinaryRequestEvents are resolved into Binary records before CrawlSetup or Snapshot hooks are executed.

Sources: README.md:55-65, TODOs.md:15-30

Plugin Ecosystem

Plugins are the fundamental units of work in abx-dl. Each plugin is a directory containing a config.json and one or more hook scripts following a specific naming convention: on_{Event}__[{order}_]{description}[.bg].{ext}.

Plugin Lifecycle Phases

Phase	Hook Type	Purpose	Records Emitted
Install	on_BinaryRequest__*	Resolves or installs required binaries (e.g., wget, chrome).	Binary
Crawl Setup	on_CrawlSetup__*	Configures shared runtime state or launches background daemons.	Machine, Process
Snapshot	on_Snapshot__*	Executes per-URL to extract content and save files.	ArchiveResult, Snapshot, Tag

Sources: README.md:60-70, abx_dl/models.py:70-110, TODOs.md:25-35

Hook Execution Logic

Hooks are treated as black-box executables. Background hooks (marked with .bg) are managed via PID files and can be either "finite" (exit on their own) or "daemon" (run until explicitly killed).

Sources: abx_dl/models.py:72-88, TODOs.md:55-65

Web Download Server

The abx-dl server provides a managed environment for running downloads as discrete sessions, which expire after 24 hours.

REST API Endpoints

Endpoint	Method	Description
/api/download	POST	Submits a URL for downloading. Returns session ID.
/api/session/<id>	GET	Retrieves status, progress records, and file listings.
/api/sessions	GET	Lists all active sessions.
/download/<id>/<path>	GET	Downloads an individual result file.
/download/<id>/zip	GET	Downloads all session output as a ZIP archive.

Sources: server/server.py:25-38, server/templates/index.html:260-310

Session Data Flow

Configuration Management

Configuration is handled through a layered approach involving environment variables, user config files, and a runtime-derived cache.

• config.env: Stores user-provided values (e.g., TIMEOUT, USER_AGENT).
• derived.env: Stores runtime-cached entries like resolved *_BINARY paths and ABX_INSTALL_CACHE.

Key Configuration Options

Variable	Type	Default	Description
TIMEOUT	Integer	60	Default timeout for hooks in seconds.
USER_AGENT	String	-	Default User-Agent string for requests.
{PLUGIN}_ENABLED	Boolean	True	Enables or disables a specific plugin.
{PLUGIN}_BINARY	Path	-	Overrides the binary path for a specific plugin.

Sources: README.md:85-115, abx_dl/cli.py:970-1000

Data Models

The system utilizes Pydantic models to enforce structure on all internal and exported data.

abx-dl serves as a lightweight, flexible gateway to powerful web archiving tools, bridging the gap between simple CLI utilities and complex archival suites.

Sources: README.md:365-370