Architecture Overview

Relevant source files

The following files were used as context for generating this wiki page:

• CLAUDE.md
• README.md
• proxybroker/api.py
• proxybroker/server.py
• proxybroker/checker.py
• proxybroker/provider_utils.py
• tests/test_server.py

ProxyBroker2 is an asynchronous framework for finding, checking, and serving public proxies. It discovers proxies from over 50 sources, validates them against judge servers, and can function as a rotating proxy server.

The system uses asyncio to manage concurrent operations across its pipeline. The architecture consists of four primary layers: Data Discovery (Providers), Validation (Checker), Resource Management (ProxyPool), and Consumption (Server/API).

Sources: CLAUDE.md:6-10, README.md:38-51

Core Components

The following table describes the primary components of the ProxyBroker2 architecture.

Component	Description	Primary File
Broker	Orchestrates the pipeline and manages provider/checker coordination.	api.py
Provider	Scrapes or fetches proxy metadata from various sources.	providers.py
Checker	Validates proxy functionality and anonymity levels.	checker.py
ProxyPool	Manages a prioritized queue of validated proxies.	server.py
Server	Provides a local proxy interface with automatic rotation.	server.py

Sources: CLAUDE.md:73-95, proxybroker/init.py:67-73

High-Level Data Flow

The diagram below illustrates how proxy data moves from discovery to the end user.

The diagram shows the transition from raw proxy discovery to validated pool management and final consumption.
Sources: CLAUDE.md:73-80, tests/test_server.py:23-28

Discovery Layer: Providers

Providers are responsible for extracting proxy IP addresses and ports from external sources. The system supports multiple provider types, including bundled sources and custom user-defined configurations.

• Provider Base Class: Implements get_proxies() to extract data.
• Concurrency Control: Limits concurrent provider tasks to MAX_CONCURRENT_PROVIDERS = 3.
• Custom Providers: Users can define providers via YAML, JSON, or Python modules.

Sources: CLAUDE.md:83-87, proxybroker/provider_utils.py:16-30

Provider Types

The provider_utils.py module defines specialized classes for different data formats:

1. SimpleProvider: Parses text, CSV, and simple JSON lists.
2. PaginatedProvider: Handles sites with numbered pages using URL templates.
3. APIProvider: Interacts with JSON APIs using authentication headers and dotted path navigation.

Sources: proxybroker/provider_utils.py:16-160, proxybroker/provider_utils.py:228-300

Validation Layer: Checker

The Checker validates whether a discovered proxy is functional and identifies its anonymity level. It performs protocol-specific negotiations and compares the proxy's perceived IP against the user's real IP to detect leaks.

The sequence shows the validation process where the Checker uses a Judge to verify the proxy's behavior.
Sources: CLAUDE.md:89-93, proxybroker/checker.py:126-170

Anonymity Levels

The Checker assigns one of three levels based on header analysis and IP matching:

• Transparent: The proxy leaks the user's real IP address in headers or the request body.
• Anonymous: The proxy does not leak the real IP but reveals it is a proxy (e.g., via Via or Proxy-Connection headers).
• High (Elite): The proxy does not leak the real IP and does not reveal that it is a proxy.

Sources: tests/test_checker.py:84-100, tests/test_server.py:116-120

Management Layer: ProxyPool

The ProxyPool manages the collection of validated proxies using a min-heap priority queue. It prioritizes proxies based on their avg_resp_time.

Pool Structure

The pool maintains two internal stages for proxies:

1. _newcomers: Proxies with fewer than min_req_proxy requests.
2. _pool: Proven proxies that satisfy quality thresholds (error rate and response time).

Sources: CLAUDE.md:78-81, proxybroker/server.py:45-80

Priority Logic

# Heap Invariant: Always use heapq operations
heapq.heappush(self._pool, (proxy.avg_resp_time, proxy))

Sources: CLAUDE.md:104-106

Parameter	Type	Default	Description
min_req_proxy	int	5	Requests needed to move from newcomers to main pool.
max_error_rate	float	0.5	Maximum allowed error rate before removal.
max_resp_time	int	8	Maximum allowed average response time in seconds.

Sources: proxybroker/server.py:45-60, tests/test_public_contracts.py:155-165

Consumption Layer: Server

The Server component provides a local proxy interface (default 127.0.0.1:8888) that rotates through the validated ProxyPool.

• Rotation: Automatically selects the "best" proxy (lowest response time) from the heap.
• Failover: If a proxy fails during a request, the server can retry with a different proxy.
• Protocol Support: Handles HTTP, HTTPS (CONNECT), and SOCKS protocols.
• Control API: Provides internal endpoints at http://proxycontrol/ for history and pool management.

Sources: CLAUDE.md:95-98, README.md:188-210, tests/test_server.py:130-150

Protocol Support and IPv6

ProxyBroker2 implements first-class IPv6 support across the entire stack.

• Canonicalization: All IPs are converted to RFC 5952 canonical form for consistent comparison.
• Protocol Priority: deterministic order: SOCKS5 → SOCKS4 → CONNECT:80 → CONNECT:25 → HTTPS → HTTP.
• SOCKS5: Supports ATYP=0x04 for IPv6 destinations.
• URI Authority: IPv6 hosts are bracketed (e.g., [2001:db8::1]:80) per RFC 3986.

Sources: CLAUDE.md:112-130, CHANGELOG.md:12-45

Summary

The ProxyBroker2 architecture leverages a highly concurrent, asynchronous pipeline to move proxies from untrusted sources to a high-quality, prioritized pool. By separating discovery, validation, and serving, the system ensures that users interact with a stable, high-performance proxy interface regardless of the volatility of the underlying public proxy sources.