Custom Providers: YAML/JSON Configurations

Relevant source files

The following files were used as context for generating this wiki page:

• proxybroker/provider_utils.py
• examples/custom_providers/using_provider_directory.py
• examples/custom_providers/simple_provider_example.py
• examples/custom_providers/paginated_provider_example.py
• examples/custom_providers/api_provider_example.py
• README.md
• CLAUDE.md

Custom Providers allow you to add new proxy sources to ProxyBroker2 without modifying the internal codebase. You can define these sources using YAML or JSON configuration files that describe how to fetch, paginate, and parse proxy data from external URLs or APIs.

The system uses specialized utility classes to handle different source types, including simple text lists, paginated websites, and structured JSON APIs. These configurations can be loaded individually or discovered automatically from specific directories, making them ideal for Docker environments where you can mount a directory of configurations as a volume.

Sources: README.md:214-230, CLAUDE.md:154-165, proxybroker/provider_utils.py:1-12

Core Provider Architecture

The custom provider system relies on the ConfigurableProvider factory class to instantiate the correct provider type based on the configuration's type field.

The diagram shows how a configuration file is dispatched through the ConfigurableProvider factory to specialized subclasses that extract proxies for the Broker.

Provider Types and Behaviors

Type	Class	Description
simple	SimpleProvider	Handles plain text lists (IP:PORT), CSV, and simple JSON/HTML formats.
paginated	PaginatedProvider	Manages sources that require fetching multiple pages via URL parameters or placeholders.
api	APIProvider	Interfaces with structured APIs requiring headers, authentication, or specific JSON path navigation.

Sources: proxybroker/provider_utils.py:348-386, proxybroker/provider_utils.py:18-20

Configuration Schema

Every configuration file must include a url and a type. Additional fields customize how the provider interacts with the source.

Common Parameters

These parameters apply to most provider types defined in the configuration.

Parameter	Type	Default	Description
url	String	(Required)	The target URL for fetching proxies.
type	String	simple	The provider class to use (simple, paginated, api).
protocols	List	[]	List of supported protocols (e.g., ["HTTP", "HTTPS"]).
max_connections	Integer	4	Maximum concurrent connections for this provider.
timeout	Integer	20	Timeout in seconds for requests.

Sources: proxybroker/provider_utils.py:361-385, proxybroker/provider_utils.py:480-515

Specialized Parameters

SimpleProvider Options

• format: Explicitly sets the parser (text, json, csv, html). If set to auto, the provider detects the format based on content.
• pattern: A custom regex string for extracting proxies. If the regex contains two capture groups, they are treated as (host, port).

PaginatedProvider Options

• page_param: The query parameter name for pagination (e.g., page or offset).
• start_page: The first page number to request.
• max_pages: Total number of pages to fetch.
• page_step: The increment between pages.

APIProvider Options

• api_key: Authorization token (sent via Authorization: Bearer, X-API-Key, or apikey headers).
• headers: Dictionary of custom HTTP headers.
• proxy_path: Dotted path to the proxy list within a nested JSON response (e.g., data.results).

Sources: proxybroker/provider_utils.py:27-33, proxybroker/provider_utils.py:181-190, proxybroker/provider_utils.py:255-263

Automated Loading and Discovery

ProxyBroker2 can automatically load all valid YAML and JSON configurations from a directory. This is controlled via the load_provider_configs_from_directory function or the --provider-dir CLI flag.

The sequence shows the process of discovering and instantiating custom providers from a local directory before they are handed to the Broker.

Directory Loading Rules

1. File Types: Only files ending in .yaml, .yml, or .json are processed.
2. Exclusions: Files starting with an underscore (e.g., _disabled.yaml) are ignored.
3. Security: The configuration loader (load_provider_configs_from_directory) only parses data; it does not execute code.
4. Python Providers: Loading .py files requires using load_python_providers_from_directory, which is opt-in and not enabled by default via CLI for security reasons.

Sources: proxybroker/provider_utils.py:390-410, proxybroker/provider_utils.py:413-424, proxybroker/provider_utils.py:438-450, CLAUDE.md:164-165

Usage Examples

YAML Configuration Example

This example shows a simple text-list provider configuration.

name: My Internal Source
type: simple
url: https://internal.example.com/proxies.txt
format: text
protocols: [HTTP, HTTPS]
timeout: 30

Sources: README.md:218-223

Loading from Python

You can pass the provider directory directly to the Broker constructor.

from proxybroker import Broker

# Load custom providers from a directory, keeping default providers
broker = Broker(provider_dirs=['/path/to/my_configs'])

# Load ONLY custom providers (disable bundled defaults)
broker_only_custom = Broker(providers=[], provider_dirs=['/path/to/my_configs'])

Sources: examples/custom_providers/using_provider_directory.py:33-43

Conclusion

The Custom Provider system enables flexible extension of proxy sources via simple data files. By abstracting the complexities of HTTP requests and parsing into SimpleProvider, PaginatedProvider, and APIProvider, ProxyBroker2 allows users to integrate new sources with minimal configuration while maintaining security through data-only loading by default.