Proxy Pool & Queue Management

Relevant source files

The following files were used as context for generating this wiki page:

• proxybroker/api.py
• proxybroker/server.py
• proxybroker/providers.py
• proxybroker/provider_utils.py
• README.md
• CLAUDE.md

The Proxy Pool and Queue Management system coordinates the discovery, validation, and distribution of proxies. It uses asynchronous queues to pass proxy data between discovery providers, the validation checker, and the end-user or rotating proxy server.

This system ensures that only proxies meeting specific quality thresholds—such as response time and error rate—are utilized in production environments. It maintains a priority-based pool to serve the "best" available proxy for any given request.

Architecture & Data Flow

The architecture relies on asyncio.Queue for inter-component communication. The Broker class orchestrates the flow, moving proxies from Provider instances into a central queue. If validation is enabled, the Checker validates these proxies before they reach the final output or the ProxyPool.

The diagram shows the progression of a proxy from discovery to prioritized usage in the server.
Sources: proxybroker/api.py:100-150, proxybroker/server.py:27-50, CLAUDE.md:65-80

The ProxyPool Component

The ProxyPool class manages a collection of validated proxies, specifically for use in the rotating proxy server mode. It implements a two-stage storage strategy to handle both unproven "newcomers" and established, high-performing proxies.

Pool Storage Mechanisms

The pool categorizes proxies based on their request history:

• Newcomers List (_newcomers): Stores proxies with fewer than min_req_proxy requests. This allows the system to gather initial performance data without affecting the "best" proxy selection.
• Main Pool Heap (_pool): Stores proven proxies in a min-heap. The heap uses proxy.avg_resp_time as the priority key, ensuring the fastest proxies are at the top.

Sources: proxybroker/server.py:30-45, CLAUDE.md:82-85

Selection Strategy

The pool currently supports the "best" strategy. When a consumer requests a proxy for a specific scheme (e.g., HTTPS), the pool performs the following logic:

1. If the total count is below min_queue, it imports more from the Broker queue.
2. If newcomers are available, it returns the first newcomer.
3. Otherwise, it pops the best proxy from the heap that supports the requested scheme.

The flow describes the prioritized selection logic within ProxyPool.get().
Sources: proxybroker/server.py:52-80

Queue Management in Broker

The Broker class acts as the primary interface for queue management. It initializes an asyncio.Queue and monitors it during find, grab, and serve operations.

Concurrent Provider Parsing

To prevent resource exhaustion, the system limits the number of concurrent providers being parsed.

• MAX_CONCURRENT_PROVIDERS: Set to 3. This defines how many provider tasks the broker executes simultaneously.
• GRAB_PAUSE: Set to 180 seconds. This is the delay between discovery cycles in server mode.

Sources: proxybroker/api.py:30-35, proxybroker/api.py:355-380

Throttling and Limits

The Broker manages the checking queue size using a semaphore-like pattern via _on_check = asyncio.Queue(maxsize=max_conn). This ensures the number of concurrent validation tasks does not exceed the max_conn configuration.

Sources: proxybroker/api.py:110-120, proxybroker/api.py:433-455

Configuration Parameters

The behavior of the pool and queues is controlled by several key parameters defined in the Broker and ProxyPool constructors.

Parameter	Type	Default	Description
max_conn	int	200	Maximum concurrent proxy checks.
min_queue	int	5	Minimum proxies required in pool before importing more.
min_req_proxy	int	5	Requests needed before a proxy moves from newcomers to the main heap.
max_error_rate	float	0.5	Maximum allowed error percentage (e.g., 0.5 = 50%).
max_resp_time	int	8	Maximum average response time in seconds.
strategy	str	'best'	Selection strategy (currently only 'best' is supported).

Sources: proxybroker/api.py:70-95, proxybroker/server.py:30-45

Proxy Removal and Maintenance

Proxies are removed from the pool if they fail to meet quality thresholds after the initial min_req_proxy period. The ProxyPool.put() method evaluates the error_rate and avg_resp_time before re-inserting a proxy into the heap.

If a specific proxy is identified as problematic by an external consumer, the ProxyPool.remove(host, port) method performs a heap-safe removal. Because heapq does not support direct removal, the system pops all items, filters the target, and restores the heap invariant, resulting in $O(N \log N)$ complexity.

Sources: proxybroker/server.py:100-145, CLAUDE.md:180-182

Summary

Proxy Pool and Queue Management provides the backbone for reliable proxy rotation. By using a two-stage pool (Newcomers vs. Heap) and strict quality filtering based on real-time performance stats, the system ensures that automated requests use the most responsive and stable proxies available. The asynchronous design allows discovery, validation, and serving to occur concurrently without blocking the main event loop.