HTTP Request Manager
Relevant source files
The following files were used as context for generating this wiki page:
The HTTP Request Manager is a core utility within the stolgo library designed to handle network communications with various financial data providers. It provides a robust abstraction layer over the Python requests library, implementing standardized timeout behaviors, retry logic, and specialized handling for both standard HTTP sessions and shell-based curl operations.
This module ensures that data retrieval from external APIs—such as Samco and Nasdaq—remains resilient against transient network failures and consistent in its connection parameters. By centralizing request logic, it allows higher-level data modules to focus on parsing and financial analysis rather than low-level network management. Sources: lib/stolgo/request.py:1-10, lib/stolgo/samco.py:16-18, lib/stolgo/nasdaq.py:14-17
Core Components and Architecture
The HTTP Request Manager is structured around three primary mechanisms: a custom HTTPAdapter for timeout management, a high-level RequestUrl class for session-based requests, and a Curl class for fallback or specialized shell-based requests.
Class Hierarchy and Relationships
The following diagram illustrates the relationship between the request management classes and the external data providers.
Detailed Logic and Implementation
Timeout Management
Standard requests.Session objects do not enforce a default timeout, which can lead to hanging processes. The TimeoutHTTPAdapter overrides the send method to ensure a DEFAULT_TIMEOUT (defaulting to 5 seconds) is applied to all requests if not explicitly provided. Sources: lib/stolgo/request.py:7-14, lib/stolgo/request.py:21-25
Retry Strategy
The RequestUrl class utilizes urllib3.util.Retry to handle server-side errors and rate limiting. The configuration includes:
- Total Retries: Defaults to 2.
- Backoff Factor: Set to 1, facilitating an exponential sleep interval between attempts.
- Status Forcelist: Automatically retries on HTTP status codes 429 (Too Many Requests), 500, 502, 503, and 504.
Sources: lib/stolgo/request.py:75-84
Shell-based Requests (Curl)
For environments where standard Python requests might be restricted or require specific shell-level headers (e.g., "cloud_mode" in Nasdaq data retrieval), the Curl class utilizes subprocess to execute system curl commands. It wraps the resulting text in a DummyResponse to maintain a compatible interface with the requests library. Sources: lib/stolgo/request.py:30-67, lib/stolgo/nasdaq.py:44-48
Data Flow for External Providers
The request manager is utilized by provider-specific classes to build and execute authenticated requests.
Request Execution Flow
Configuration and Defaults
The behavior of the HTTP Request Manager is governed by several constants and parameters passed during initialization.
| Parameter | Type | Default | Description |
|---|---|---|---|
DEFAULT_TIMEOUT | int | 5 | Time in seconds to wait for a server response. |
MAX_RETRIES | int | 2 | Number of attempts for failed requests. |
backoff_factor | int | 1 | Multiplier for delay between retries. |
status_forcelist | list | [429, 500, 502, 503, 504] | HTTP codes that trigger an automatic retry. |
Sources: lib/stolgo/request.py:7-8, lib/stolgo/request.py:78-82
Error Handling
The manager handles exceptions by raising descriptive messages when HTTP errors occur. It specifically uses page.raise_for_status() to catch non-200 range responses and wraps them in custom exception messages including the error content. Sources: lib/stolgo/request.py:91-95, lib/stolgo/request.py:100-104
Summary
The HTTP Request Manager provides the network backbone for stolgo. By implementing the TimeoutHTTPAdapter and providing a dual-path request execution (Requests/Curl), it ensures that financial data fetching remains stable across different execution environments and network conditions. This infrastructure supports higher-level operations like resourcing historical stock data and fetching real-time option chains. Sources: lib/stolgo/request.py:1-105, setup.py:21-28