HTTP Control API
Relevant source files
The following files were used as context for generating this wiki page:
The HTTP Control API provides a management interface for the ProxyBroker2 server during runtime. It allows users to query proxy usage history and modify the active proxy pool through standard HTTP requests.
The API functions by intercepting requests directed at a pseudo-hostname called proxycontrol. When the Server detects this host in the Host header, it processes the request internally rather than forwarding it to an external proxy.
Sources: proxybroker/server.py:192-194, README.md:424-426
Architecture and Request Interception
The control API resides within the Server._handle method. This method processes every incoming connection to the local proxy server.
Interception Logic
The server identifies API calls by checking the Host header of the incoming request. If the header matches proxycontrol, the server stops the standard proxying flow. It then splits the request path to determine the requested operation and parameters.
The diagram shows the decision logic used by the server to distinguish between standard traffic and administrative API calls.
Sources: proxybroker/server.py:192-196, tests/test_server.py:145-148
API Endpoints
The API supports two primary operations: removing proxies and retrieving usage history.
Endpoint Summary
| Operation | Path | Description | Result |
|---|---|---|---|
| Remove | /api/remove/<IP>:<PORT> | Removes a specific proxy from the current ProxyPool. | 204 No Content |
| History | /api/history/url:<URL> | Retrieves the IP:PORT of the proxy used for a specific URL. | JSON Object |
Sources: proxybroker/server.py:196-222, README.md:434-469
Proxy Removal
The remove operation triggers the ProxyPool.remove(host, port) method. This method searches both the _newcomers list and the prioritized _pool heap. To maintain data integrity, it uses a heap-safe removal process that preserves priority queue invariants.
Sources: proxybroker/server.py:117-142, proxybroker/server.py:197-207
History Retrieval
The history operation queries a TTLCache named history. This cache stores mappings between client-request combinations and the specific proxy assigned to them.
- Key Composition: The cache key combines the client's peer IP address and the requested URL.
- HTTPS Limitation: For encrypted traffic, the server only records the hostname and port (e.g.,
httpbin.org:443) because it cannot inspect encrypted request paths.
Sources: proxybroker/server.py:31, proxybroker/server.py:208-222, README.md:452-454
Data Management
History Cache
Usage data is stored in a time-to-live (TTL) cache to prevent memory exhaustion while providing recent audit logs.
| Attribute | Value | Source |
|---|---|---|
| Storage Type | cachetools.TTLCache | proxybroker/server.py:31 |
| Max Size | 10,000 entries | proxybroker/server.py:31 |
| TTL | 600 seconds (10 minutes) | proxybroker/server.py:31 |
Response Injection
While the Control API handles administrative queries, the server also injects proxy metadata into standard HTTP responses. It adds the X-Proxy-Info header to the response, containing the IP:PORT of the proxy used for that specific request.
Sources: proxybroker/server.py:245-247, README.md:394-396
Sequence of API Interaction
The sequence diagram illustrates the workflow for identifying a failing proxy through the history API and subsequently removing it from the pool.
Sources: proxybroker/server.py:192-222, tests/test_integration.py:46-55
Conclusion
The HTTP Control API enables dynamic management of the ProxyBroker2 server without requiring restarts or external configuration changes. By using the proxycontrol pseudo-host and a dedicated TTL cache, it provides developers with tools to monitor proxy assignments and prune underperforming nodes from the rotation.