Development Setup
Relevant source files
The following files were used as context for generating this wiki page:
abx-dl is an all-in-one CLI tool designed to download and extract content from URLs using the ArchiveBox plugin ecosystem. It provides a simplified interface to various downloading and scraping tools like wget, puppeteer, and yt-dlp. Development requires a Python 3.11+ environment and utilizes uv for dependency management and environment isolation.
The project architecture relies on a hook-based system where plugins execute in distinct phases: Install, BinaryRequest, CrawlSetup, and Snapshot. Developers contribute to this ecosystem by creating or modifying plugins that adhere to these execution phases.
Sources: pyproject.toml:4-6, README.md:1-12, README.md:41-50
Environment Preparation
The project uses uv as the primary build and dependency management tool. The development environment requires Python 3.11 or newer.
Initial Installation
To set up the local development environment, use the following commands:
# Install dependencies including dev group
uv sync --extra dev
# Activate the virtual environment
source .venv/bin/activate
Sources: CLAUDE.md:12-16, pyproject.toml:6-7
Dependency Groups
The project defines specific dependency sets in pyproject.toml.
| Group | Key Dependencies | Purpose |
|---|---|---|
| Project | rich-click, rich, abxbus, abx-pkg, pydantic, requests | Core CLI and logic |
| Optional (dev) | flask, pytest, pytest-xdist, ruff, flake8, mypy | Testing and Linting |
| Dependency-groups | flask, prek, pyright, ruff, ty | Typing and IDE support |
Sources: pyproject.toml:45-77, pyproject.toml:132-138
Plugin Management and Architecture
abx-dl is built on a plugin-based architecture. Plugins are discovered from a specific directory and registered as Plugin models containing metadata and Hook executables.
Plugin Discovery Flow
The system searches for plugins in a directory specified by the ABX_PLUGINS_DIR environment variable or the default location provided by the abx-plugins package.
Hook Naming Convention
Hooks are executable scripts within plugin directories. Their behavior is determined by their filename structure:
on_{Event}__[{order}_]{description}[.bg].{ext}
- Event: The bus event family (Install, BinaryRequest, CrawlSetup, Snapshot).
- Order: Optional integer for execution priority.
- bg: Marks the hook as a background process.
Sources: abx_dl/models.py:35-50, abx_dl/models.py:330-344
Development Workflow
Running the Tool Locally
Developers can run the tool directly from the source using uv run.
# Run against a URL with specific plugins
uv run abx-dl dl --plugins=chrome,title 'https://example.com'
# Show current configuration
uv run abx-dl config
Sources: CLAUDE.md:21-23, abx_dl/cli.py:596-610
Configuration Layers
Configuration is handled through environment variables and local files.
| File Location | Purpose |
|---|---|
~/.config/abx/config.env | User-provided persistent values (via config --set) |
~/.config/abx/derived.env | Runtime-derived cache (binary paths, install timestamps) |
Sources: README.md:54-68, README.md:92-96
Testing and Quality Assurance
Testing is conducted using pytest. The project includes tests for CLI behavior, model integrity, and plugin execution.
# Run all tests
uv run pytest tests/
# Run linting and formatting
uv run ruff check abx_dl/
uv run ruff format abx_dl/
uv run mypy abx_dl/
Sources: CLAUDE.md:32-44
Web Download Server
For testing the web interface and REST API, a Flask-based server is provided in server/server.py.
Server Setup
uv run server/server.py --host 0.0.0.0 --port 8484 --data-dir /tmp/abx-dl-sessions
Sources: server/server.py:16-18
API Interaction Sequence
The server manages asynchronous downloads by spawning abx-dl processes.
Release Process
The project follows a standard versioning and release workflow:
- Version Bump: Update
versioninpyproject.toml. - Tagging: Commit changes and create a git tag (e.g.,
v1.10.21). - CI/CD: Pushing a tag triggers the
.github/workflows/publish.ymlworkflow, which publishes to PyPI using a trusted publisher.
Sources: CLAUDE.md:46-59, pyproject.toml:5