Proxy Server Mode: Rotation & Failover
Relevant source files
The following files were used as context for generating this wiki page:
Proxy Server Mode operates a local proxy server that distributes incoming client requests to a pool of discovered external proxies. The system automatically rotates proxies to optimize performance and implements failover logic to discard failing or slow proxies.
Sources: proxybroker/server.py:126-128, README.md:27-28
Architecture Overview
The system consists of three main internal components: the Broker, the Server, and the ProxyPool. The Broker finds and validates proxies, feeding them into a queue. The Server listens for client connections and uses the ProxyPool to select the most suitable proxy for each request.
The diagram shows the data flow from proxy discovery to client request fulfillment.
Sources: proxybroker/api.py:321-326, proxybroker/server.py:40-42
ProxyPool Management & Rotation
The ProxyPool manages the selection and health of proxies using a priority queue (min-heap) based on avg_resp_time. It separates proxies into "newcomers" and an established pool to ensure new proxies are tested before full integration.
Selection Strategy
The ProxyPool currently supports the "best" strategy. It pops the proxy with the lowest average response time from the heap. If the selected proxy does not support the required protocol (e.g., HTTPS), it is temporarily set aside until a suitable one is found or a new one is imported from the Broker queue.
Sources: proxybroker/server.py:53-56, proxybroker/server.py:64-79
Rotation and Quality Control
Proxies return to the pool via the put() method, which performs quality checks. The pool discards proxies that exceed error rate or response time thresholds.
| Threshold | Description | Default Value |
|---|---|---|
max_error_rate | Maximum allowed percentage of failed requests (0.5 = 50%) | 0.5 |
max_resp_time | Maximum allowed average response time in seconds | 8 |
min_req_proxy | Minimum requests required before quality enforcement starts | 5 |
Sources: proxybroker/server.py:44-50, proxybroker/server.py:108-117
Failover Logic
Failover occurs during the request handling phase in the Server. If a proxy fails to connect or returns errors, the Server catches the exception and attempts the request again using a different proxy from the pool, up to a defined max_tries limit.
The sequence shows the failover mechanism when the first selected proxy fails.
Sources: proxybroker/server.py:273-305, tests/test_server.py:126-141
Proxy Control API
The Server exposes a special internal API via the proxycontrol hostname. Clients can send standard HTTP requests to this host to interact with the proxy pool state.
Endpoints
| Path | Description | Result |
|---|---|---|
http://proxycontrol/api/remove/<ip>:<port> | Removes a specific proxy from the pool manually. | 204 No Content |
http://proxycontrol/api/history/url:<url> | Retrieves the IP:Port of the proxy used for a specific URL. | 200 OK (JSON) |
Sources: proxybroker/server.py:246-271, README.md:200-246
Configuration
The server mode can be configured through the CLI using python -m proxybroker serve or via the Broker.serve() Python method.
CLI Parameters
--host: The interface to bind the server to (default:127.0.0.1).--port: The local port to listen on (default:8888).--limit: The number of working proxies to maintain in the pool.--min-queue: The minimum number of proxies to collect before starting selection.--prefer-connect: Whether to use theCONNECTmethod for HTTP requests if supported.
Sources: proxybroker/cli.py:165-225, proxybroker/api.py:218-270
# Server instantiation example from server.py
server = Server(
host=host,
port=port,
proxies=self._proxies,
timeout=self._timeout,
max_tries=kwargs.pop("max_tries", self._max_tries),
loop=self._loop,
**kwargs,
)
Sources: proxybroker/api.py:321-329
Conclusion
The Proxy Server Mode provides a resilient entry point for proxying traffic. By combining a priority-based heap for selection and active failover during request execution, it ensures that client requests are routed through the most reliable proxies available while transparently handling individual proxy failures.