Built-in Proxy Providers

Relevant source files

The following files were used as context for generating this wiki page:

• proxybroker/providers.py
• proxybroker/provider_utils.py
• proxybroker/init.py
• CLAUDE.md
• README.md
• examples/custom_providers/simple_provider_example.py
• examples/custom_providers/api_provider_example.py

Built-in Proxy Providers are modules that discover public proxies from over 50 different online sources. These providers operate asynchronously to fetch, parse, and extract proxy information, including IP addresses, ports, and supported protocols.

The system uses a base Provider class to standardize how data is retrieved from various websites. Each provider implements specific logic to handle the unique structure of its source, such as HTML tables, plain text lists, or JSON APIs.

Sources: proxybroker/providers.py:32-41, CLAUDE.md:43-46, README.md:18-20

Provider Architecture

The provider system relies on an asynchronous pipeline to fetch data without blocking the main execution thread. The Provider base class manages connection limits, retries, and session handling.

Core Components

Component	Description
Provider	The base class for all proxy sources.
PROVIDERS	A list containing all pre-configured built-in provider instances.
_sem_provider	An asyncio.Semaphore that limits concurrent connections per provider.
proxies	A set that stores unique found proxies as tuples of (host, port, proto).

Sources: proxybroker/providers.py:32-72, proxybroker/providers.py:598-662

Data Flow

The following diagram shows the lifecycle of a provider during a proxy discovery task.

The provider fetches the raw content of a page, applies regex patterns or custom parsers, and normalizes the results into a standard format.

Sources: proxybroker/providers.py:74-124

Built-in Source Implementation

Built-in providers are categorized based on their technical implementation or the platform they scrape.

Standard Web Scrapers

Most built-in providers inherit from the base Provider and use the IPPortPatternGlobal regex to find proxies in HTML or text responses. Some providers implement custom _pipe methods to navigate multiple pages or handle sessions.

• Blogspot Providers: Sources like sslproxies24.blogspot.com use a specific Blogspot_com_base class to handle the standard Blogspot layout.
• API-based Providers: Services like api.proxyscrape.com provide direct text or JSON feeds that the system parses.

Sources: proxybroker/providers.py:127-160, proxybroker/providers.py:599-609

Specialized Parsers

Some providers require complex decoding because the proxy information is obfuscated via JavaScript or custom encoding.

• Proxy_list_org: Uses Base64 decoding to extract the proxy host and port from JavaScript calls.
• Spys_ru: Implements logic to solve JavaScript-based port obfuscation by mapping characters to numerical digits.
• Free_proxy_cz: Decodes Base64 strings representing host and port values.

Sources: proxybroker/providers.py:196-200, proxybroker/providers.py:461-482, proxybroker/providers.py:522-526

Provider Utilities and Templates

The proxybroker.provider_utils module provides enhanced classes for common provider patterns, allowing users to define new sources with minimal code.

Configurable Providers

The system supports ConfigurableProvider, which creates provider instances from YAML or JSON configuration files. This allows adding new sources without modifying the Python source code.

Configuration Field	Description
type	The type of provider (simple, paginated, or api).
url	The source URL or URL template.
protocols	A list of expected protocols (e.g., ["HTTP", "HTTPS"]).
max_connections	Limits concurrent requests to the source (default: 4).

Sources: proxybroker/provider_utils.py:378-422

Extraction Logic

Providers use the find_proxies method to extract data from a page. The system handles both IPv4 and IPv6 addresses.

IPv6 Support

The system extracts bracketed IPv6 addresses using the IPv6BracketedPortPattern. Before performing the standard IPv4 pass, it masks bracketed IPv6 spans to prevent "phantom" IPv4 entries from being created from IPv4-mapped IPv6 addresses (e.g., [::ffff:192.0.2.1]).

Sources: proxybroker/providers.py:114-124, CHANGELOG.md:16-25

Parsing Formats

SimpleProvider supports automatic format detection and parsing for several common data types:

• JSON: Navigates through common wrapper keys like proxies, data, or items.
• CSV: Uses the standard csv module to handle quoted fields correctly.
• Text: Parses host:port patterns or whitespace-separated parts.

Sources: proxybroker/provider_utils.py:74-106, proxybroker/provider_utils.py:146-160

Summary

Built-in Proxy Providers form the discovery layer of ProxyBroker2. By utilizing a common interface and specialized parsers, the system can concurrently aggregate thousands of proxies from diverse sources. The inclusion of provider_utils enables the system to scale beyond its built-in list through configuration-based custom providers.

Sources: README.md:18-20, CLAUDE.md:112-118