Advanced Custom Providers: Pagination & APIs
Relevant source files
The following files were used as context for generating this wiki page:
Advanced Custom Providers allow you to extend ProxyBroker2 by defining new sources for proxy discovery through complex interfaces such as paginated websites and authenticated APIs. The system provides specialized classes to handle URL parameter substitution, authentication headers, and nested data extraction.
These providers enable the discovery of proxies from sources that require more than a single URL fetch. You can define these sources using Python classes or YAML/JSON configuration files, which the system loads automatically from specified directories.
Sources: README.md:214-239, CLAUDE.md:154-165, proxybroker/provider_utils.py:11-15
Provider Architecture and Class Hierarchy
The project uses an inheritance-based architecture for providers. Every specialized provider inherits from the base Provider class or one of the utility subclasses in proxybroker.provider_utils.
The diagram shows the relationship between different provider types and their specialized methods for data extraction and navigation.
Sources: proxybroker/provider_utils.py:18-20, proxybroker/provider_utils.py:169-173, proxybroker/provider_utils.py:254-258, proxybroker/provider_utils.py:356-358
Paginated Providers
PaginatedProvider handles sources where proxy lists span multiple pages. It automates the generation of sequential URLs based on a starting page, a maximum page count, and a step interval.
Pagination Logic
The provider generates a range of pages using range(start_page, start_page + (max_pages * page_step), page_step). It constructs the final URLs by either:
- Replacing
{}placeholders in thebase_url. - Appending or replacing the
page_paramin the URL query string.
Sources: proxybroker/provider_utils.py:188-214, tests/test_provider_utils.py:133-145
URL Construction Examples
| Source URL Type | Parameter | Result (Page 2) |
|---|---|---|
http://site.com/p-{}.html | N/A | http://site.com/p-2.html |
http://site.com/list | page | http://site.com/list?page=2 |
http://site.com/list?page=1 | page | http://site.com/list?page=2 |
Sources: proxybroker/provider_utils.py:216-241, tests/test_provider_utils.py:147-160
This flow represents how PaginatedProvider iterates through multiple pages to aggregate proxy data.
Sources: proxybroker/provider_utils.py:192-196, examples/custom_providers/paginated_provider_example.py:9-30
API Providers
APIProvider interacts with structured data endpoints. It supports common authentication methods and allows navigation through nested JSON structures to find the proxy list.
Authentication and Headers
The APIProvider copies custom_headers and injects API keys using common header names if api_key is provided:
X-API-KeyAuthorization: Bearer <key>apikey
Sources: proxybroker/provider_utils.py:277-293, examples/custom_providers/api_provider_example.py:10-23
Data Extraction Path
If the proxy list is nested within a JSON response, you use proxy_path (e.g., "data.results.items") to navigate the dictionary. The _walk_proxy_path method splits this string by dots and traverses the dictionary keys.
Sources: proxybroker/provider_utils.py:312-320, tests/test_provider_utils.py:221-235
JSON Mapping Fields
APIProvider and SimpleProvider automatically look for specific keys when parsing JSON objects:
- IP/Host Keys:
ip,host,address,proxy,proxy_ip - Port Keys:
port,proxy_port,p
Sources: proxybroker/provider_utils.py:84-88, proxybroker/provider_utils.py:339-348
Configuration and Loading
ProxyBroker2 can load these advanced providers from external files without requiring code modifications in the core package.
Configuration Template (YAML)
name: My API Source
type: api
url: https://api.example.com/v1/proxies
api_key: "${ENV_VAR}"
response_format: json
proxy_path: data.proxies
protocols: [HTTP, HTTPS]
max_connections: 2
timeout: 30
Sources: proxybroker/provider_utils.py:539-563, README.md:214-222
Loading Methods
The system distinguishes between safe data-only loading and code-executing Python loading.
| Method | File Types | Security Level |
|---|---|---|
load_provider_configs_from_directory | .yaml, .yml, .json | Safe (Data Only) |
load_python_providers_from_directory | .py | High Risk (Executes Code) |
load_providers_from_directory | All | Configurable via allow_python |
Sources: proxybroker/provider_utils.py:408-417, proxybroker/provider_utils.py:461-470, proxybroker/provider_utils.py:515-524
Summary
Advanced Custom Providers extend the discovery capabilities of ProxyBroker2 through specialized classes for pagination and API interaction. PaginatedProvider automates multi-page fetches via URL substitution, while APIProvider manages authentication and nested JSON parsing. These components can be instantiated programmatically in Python or defined in external YAML/JSON configuration files for automatic loading by the Broker.
Sources: proxybroker/provider_utils.py:11-15, proxybroker/provider_utils.py:356-365, CLAUDE.md:154-165