Custom Providers: Python API Extension

Relevant source files

The following files were used as context for generating this wiki page:

• proxybroker/provider_utils.py
• examples/custom_providers/simple_provider_example.py
• examples/custom_providers/api_provider_example.py
• examples/custom_providers/paginated_provider_example.py
• examples/custom_providers/using_provider_directory.py
• proxybroker/api.py
• tests/test_provider_utils.py

ProxyBroker2 allows you to extend the system by adding custom proxy sources through the Python API. This extension mechanism enables the integration of private lists, paid APIs, and specific web scrapers without modifying the core codebase. Sources: README.md:148-150, proxybroker/provider_utils.py:1-5

Architecture Overview

Custom providers inherit from a base Provider class or specialized utility classes. These classes handle the asynchronous fetching and parsing of proxy data from various endpoints. The Broker class orchestrates these providers by calling their find_proxies or _pipe methods during discovery cycles. Sources: proxybroker/provider_utils.py:12-16, proxybroker/api.py:38-42

Class Hierarchy

The following diagram illustrates the relationship between the base provider and specialized utility classes:

The hierarchy provides specialized logic for common web patterns like flat lists, paginated results, and JSON APIs. Sources: proxybroker/provider_utils.py:12-290

Specialized Provider Types

SimpleProvider

The SimpleProvider handles basic proxy lists such as plain text, HTML tables, JSON, or CSV files. It includes automatic format detection and supports custom regex patterns for extraction. Sources: proxybroker/provider_utils.py:12-25

Feature	Description
Format Detection	Automatically identifies text, json, csv, or html based on content.
Custom Patterns	Accepts a regex pattern for non-standard string formats.
JSON Unwrapping	Automatically extracts lists from keys like proxies, data, or items.
Colon Parsing	Extracts numeric ports while ignoring trailing comments or tags (e.g., 1.2.3.4:8080 # US).

Sources: proxybroker/provider_utils.py:53-157, tests/test_provider_utils.py:70-98

PaginatedProvider

The PaginatedProvider fetches proxies from sites that distribute lists across multiple pages. It constructs URLs by injecting page numbers into placeholders or query parameters. Sources: proxybroker/provider_utils.py:168-185

• URL Templates: Supports {} placeholders or page= parameters.
• Page Control: Configures start_page, max_pages, and page_step.
• Parameter Preservation: Replaces existing page parameters while preserving other query strings.

Sources: proxybroker/provider_utils.py:202-230, tests/test_provider_utils.py:155-190

APIProvider

The APIProvider interacts with structured JSON APIs. It supports authentication via API keys and deep navigation into nested JSON structures using dotted path notation. Sources: proxybroker/provider_utils.py:246-261

Provider Loading and Registration

Custom providers can be registered programmatically or loaded dynamically from directories. Sources: proxybroker/provider_utils.py:348-435

Directory Loading

ProxyBroker2 identifies providers in a specified directory based on file extensions:

• YAML/JSON: Loaded via ConfigurableProvider. This is safe for untrusted environments as it only reads data. Sources: proxybroker/provider_utils.py:364-375
• Python (.py): Loaded via load_python_providers_from_directory. This executes arbitrary code and should only be used for trusted local files. Sources: proxybroker/provider_utils.py:408-420

Programmatic Usage

You pass custom provider instances to the Broker constructor. If the providers argument is an empty list [], the broker ignores bundled defaults and only uses the custom sources. Sources: examples/custom_providers/using_provider_directory.py:33-40, README.md:270-280

from proxybroker import SimpleProvider, Broker

class MyProvider(SimpleProvider):
    domain = "mysite.com"
    def __init__(self):
        super().__init__(
            url="http://mysite.com/proxies.txt",
            format='text',
            proto=('HTTP', 'HTTPS')
        )

broker = Broker(providers=[MyProvider()])

Sources: README.md:270-280, examples/custom_providers/simple_provider_example.py:6-16

Configuration Options

When defining providers via YAML or JSON, the following fields control behavior:

Field	Type	Required	Description
url	string	Yes	The source URL (or template for paginated).
type	string	No	simple, paginated, or api. Default: simple.
protocols	list	No	List of supported protocols (e.g., ["HTTP", "SOCKS5"]).
max_connections	int	No	Maximum concurrent connections to this source.
timeout	int	No	Request timeout in seconds.
proxy_path	string	No	(API only) Dotted path to the proxy list in JSON.

Sources: proxybroker/provider_utils.py:295-340, examples/custom_providers/using_provider_directory.py:16-25

Summary

The Python API Extension allows for highly flexible proxy sourcing. By subclassing SimpleProvider, PaginatedProvider, or APIProvider, you can implement custom parsing logic or authentication flows. These providers integrate seamlessly into the Broker discovery pipeline, enabling the tool to gather proxies from both built-in and user-defined sources. Sources: proxybroker/provider_utils.py:1-10, README.md:255-260