Orchestrator Service

Relevant source files

The following files were used as context for generating this wiki page:

• abx_dl/orchestrator.py
• abx_dl/cli.py
• README.md
• server/server.py
• TODOs.md
• skills/abx-dl/SKILL.md

The Orchestrator Service is the central execution engine of abx-dl. It manages the lifecycle of web content extraction by coordinating plugin discovery, dependency resolution, and the execution of various hook phases. It bridges the gap between high-level user requests (from the CLI or Web UI) and the low-level execution of specialized tools like wget, yt-dlp, and headless browsers.

The service is built around an asynchronous event-driven architecture, utilizing an EventBus to propagate state changes and results. It ensures that dependencies are satisfied, environments are configured, and snapshots are captured in a structured, reproducible manner. Sources: abx_dl/orchestrator.py, abx_dl/cli.py:535-544, README.md:129-136

Core Architecture and Lifecycle

The orchestration process follows a strictly defined sequence of phases to ensure all necessary tools are available and configured before extraction begins. This lifecycle is managed via the download() function, which acts as the primary entry point for the service. Sources: abx_dl/orchestrator.py, TODOs.md:19-27

Execution Phases

Phase	Description
Install Phase	Identifies required_binaries from plugins and triggers BinaryRequestEvents.
Binary Resolution	Resolves or installs required binaries via on_BinaryRequest__* hooks.
CrawlSetup Phase	Launches daemons or shared runtime state via on_CrawlSetup__* hooks.
Snapshot Phase	Executes per-URL extraction logic using on_Snapshot__* hooks.
Cleanup Phase	Terminates background processes and performs final resource cleanup.

Sources: README.md:46-52, TODOs.md:32-44, abx_dl/orchestrator.py

Lifecycle Flow Diagram

The following diagram illustrates the transition between high-level events and plugin hook execution managed by the orchestrator.

Component Roles

Orchestrator Functions

The abx_dl/orchestrator.py module contains the logic for managing these phases:

• download(): Coordinates the full sequence from installation to cleanup for a given URL.
• install_plugins(): Specifically handles the dependency resolution layer.
• compute_phase_timeout(): Calculates total timeouts based on the specific hooks being executed.
• create_bus(): Initializes the EventBus used for internal communication.

Sources: abx_dl/orchestrator.py, abx_dl/cli.py:535-544

Event System

The orchestrator relies on structured events to track progress and communicate with UIs (CLI/Web).

• Process Events: ProcessStartedEvent, ProcessStdoutEvent, and ProcessCompletedEvent track the execution of hook subprocesses.
• Domain Events: BinaryEvent (resolved binaries), SnapshotEvent (URL tracking), and ArchiveResultEvent (extracted content).
• Control Events: CrawlPauseEvent and CrawlAbortEvent allow for runtime intervention.

Sources: abx_dl/cli.py:56-69, abx_dl/cli.py:657-675

Dependency Management (Binary Resolution)

The orchestrator manages dependencies by mapping plugin requirements to binary providers. If a required binary (e.g., chromium) is missing, the orchestrator emits a BinaryRequestEvent. Provider plugins (like npm, pip, or apt) listen for these events and attempt to resolve or install the binary. Sources: README.md:129-136, TODOs.md:48-57

Integration with Services

CLI Interface

The CLI (abx_dl/cli.py) utilizes the orchestrator via a LiveBusUI class, which renders real-time tables of active processes and results. It maps orchestrator events to a "Rich" console display. Sources: abx_dl/cli.py:535-560, abx_dl/cli.py:586-610

Web Server

The Web Download Server (server/server.py) wraps the orchestrator's CLI interface in a background thread for each session. It persists session metadata in session.json and parses index.jsonl (emitted by the orchestrator) to provide status updates via a REST API. Sources: server/server.py:126-140, server/server.py:209-219

Configuration Handling

The orchestrator utilizes a layered configuration system:

1. User Config: Persistent settings in config.env.
2. Derived Cache: Runtime-resolved paths (like CHROME_BINARY) stored in derived.env.
3. Environment Variables: Temporary overrides (e.g., TIMEOUT=120).

Sources: README.md:58-75, skills/abx-dl/SKILL.md:68-84

Summary

The Orchestrator Service is the core logic engine of abx-dl, providing a robust framework for multi-phase web extraction. By decoupling binary provision from content extraction and utilizing an event-driven model, it allows for a highly extensible plugin ecosystem that can be consumed by both terminal-based users and web-based API clients. Sources: abx_dl/orchestrator.py, README.md:125-136, server/server.py:27-38