Relevant source files

The following files were used as context for generating this wiki page: - [server/server.py](https://github.com/ArchiveBox/abx-dl/blob/main/server/server.py) - [server/templates/index.html](https://github.com/ArchiveBox/abx-dl/blob/main/server/templates/index.html) - [server/templates/session.html](https://github.com/ArchiveBox/abx-dl/blob/main/server/templates/session.html) - [abx_dl/cli.py](https://github.com/ArchiveBox/abx-dl/blob/main/abx_dl/cli.py) - [README.md](https://github.com/ArchiveBox/abx-dl/blob/main/README.md)

# Web Server & UI Components The `abx-dl` Web Server is a Flask-based application that provides a graphical user interface and a REST API for interacting with the `abx-dl` download engine. It allows users to submit URLs for processing, monitor progress in real-time, and retrieve archived files through a web browser or programmatic HTTP requests. Sources: [server/server.py:1-25](), [README.md:320-322]() The system is designed to manage "sessions," where each download request is isolated in its own directory. It handles the orchestration of background processes, session lifecycle management (including a 24-hour expiration policy), and the presentation of structured logs and progress records derived from the underlying plugin execution. Sources: [server/server.py:27-35](), [server/templates/index.html:260-262]() ## System Architecture The web component operates as a wrapper around the `abx-dl` CLI. When a request is received, the server spawns a background thread that executes the `abx-dl` command-line tool, capturing its output and metadata into a session-specific directory. Sources: [server/server.py:108-115](), [server/server.py:133-145]() ```mermaid graph TD Client[Web Browser / API Client] -->|HTTP POST| Flask[Flask Server] Flask -->|Spawn Thread| BgThread[Background Worker] BgThread -->|Execute| CLI[abx-dl CLI Process] CLI -->|Write| Filesystem[Session Data Dir] Filesystem -->|Read Status/Files| Flask Flask -->|Stream Logs/UI| Client ``` The diagram illustrates the flow from a client request to the execution of the CLI and subsequent data persistence. Sources: [server/server.py:100-150](), [server/server.py:218-230]() ### Session Lifecycle Management Sessions are the primary unit of work. Each session is assigned a unique 12-character hexadecimal ID. The server maintains an in-memory store of active sessions while also persisting metadata to a `session.json` file in the session directory to allow for recovery after a server restart. Sources: [server/server.py:84-106](), [server/server.py:186-198]() | Feature | Description | | :--- | :--- | | **Isolation** | Each request gets a unique directory in the data directory. | | **Persistence** | Metadata is saved in `session.json`; results are in `index.jsonl`. | | **Backgrounding** | Downloads run in separate threads to prevent blocking the web server. | | **Cleanup** | A background thread removes sessions older than 24 hours. | Sources: [server/server.py:84-100](), [server/server.py:146-160](), [server/server.py:260-270]() ## REST API Specification The server exposes a RESTful API for programmatic control over the download process. It supports submission, status polling, and file retrieval. Sources: [server/server.py:320-440](), [server/templates/index.html:198-210]() ### API Endpoints | Method | Endpoint | Description | | :--- | :--- | :--- | | `POST` | `/api/download` | Submits a URL and list of plugins for processing. | | `GET` | `/api/sessions` | Lists all active and recent sessions. | | `GET` | `/api/session/` | Returns detailed status, file list, and progress records. | | `GET` | `/download//` | Downloads a specific result file. | | `GET` | `/download//zip` | Downloads all session results as a ZIP archive. | Sources: [server/server.py:320-438](), [server/templates/index.html:198-235]() ### Data Flow for API Download ```mermaid sequenceDiagram participant User as API Client participant API as Flask API participant Worker as Background Process User->>API: POST /api/download {url, plugins} API->>API: Create Session ID API->>Worker: Start abx-dl dl --output API-->>User: 201 Created {id, status: starting} loop Polling User->>API: GET /api/session/ API-->>User: {status: running, progress: [...]} end Worker->>API: Process Completed (exit 0) User->>API: GET /download//zip API-->>User: abx-dl-session.zip ``` This sequence shows the typical lifecycle of an API-driven download from submission to final file retrieval. Sources: [server/server.py:320-360](), [server/templates/index.html:240-255]() ## UI Components & Templates The Web UI is composed of two primary pages: the **Index** (Submission) page and the **Session** (Detail) page. Sources: [server/templates/index.html](), [server/templates/session.html]() ### Submission Interface (`index.html`) The homepage provides a form for users to enter a URL and select from a list of `ALLOWED_PLUGINS`. It includes JavaScript helpers to select default plugins, all plugins, or clear the selection. Sources: [server/templates/index.html:170-195](), [server/templates/index.html:268-280]() ### Session Detail Interface (`session.html`) The session page provides real-time monitoring of a download. It uses auto-refresh (every 3 seconds) while the session status is 'starting' or 'running'. Sources: [server/templates/session.html:565-568]() Key components include: * **Progress List:** Displays records parsed from `index.jsonl`, showing plugin execution and hook status. Sources: [server/templates/session.html:515-535]() * **Result Files:** A list of generated files with links for individual download and a button to download everything as a ZIP. Sources: [server/templates/session.html:542-555]() * **Logs:** Toggleable sections showing the raw `stdout` and `stderr` from the `abx-dl` process. Sources: [server/templates/session.html:572-585]() ## Configuration & Plugin Integration The server limits exposed plugins to a safe subset defined in `ALLOWED_PLUGINS` to prevent system-level plugins from being triggered via the public web interface. Sources: [server/server.py:65-98]() ```python ALLOWED_PLUGINS = [ "title", "favicon", "screenshot", "pdf", "dom", "wget", "singlefile", "readability", "headers", "hashes", "git", "ytdlp", "gallerydl", "forumdl", "mercury", "htmltotext", "trafilatura", "defuddle", "media", "archivedotorg", "dns", "ssl", "seo", "accessibility", "redirects", "responses", ] ``` Sources: [server/server.py:65-95]() ### Data Directory Structure The server uses a structured directory format to store session data. Sources: [README.md:250-265](), [server/server.py:218-240]() ```text /tmp/abx-dl-sessions// ├── session.json # Metadata (URL, status, timestamps) ├── index.jsonl # ArchiveBox-compatible result manifest ├── abx-dl.stdout.log # Main process stdout ├── abx-dl.stderr.log # Main process stderr └── / # Subdirectory for plugin-specific outputs └── output_file.ext ``` Sources: [server/server.py:133-145](), [README.md:250-265]() ## Conclusion The Web Server & UI Components provide a robust, asynchronous interface for the `abx-dl` tool. By utilizing background threading and structured logging, the system ensures that long-running downloads do not impact server responsiveness while providing users with comprehensive visibility into the archiving process via both a modern web interface and a standard REST API. Sources: [server/server.py:1-40](), [README.md:320-325]()