REST API Streams

Relevant source files

The following files were used as context for generating this wiki page:

• singer_sdk/streams/rest.py
• singer_sdk/streams/graphql.py
• packages/meltano-tap-gitlab/tap_gitlab/streams.py
• tests/core/rest/test_failure.py
• cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md
• CLAUDE.md

REST API Streams provide a structured framework within the Meltano Singer SDK for extracting data from HTTP-based services. These streams inherit from the RESTStream class, which automates common API interaction tasks such as request construction, authentication, pagination handling, and response parsing. By utilizing this framework, developers can focus on endpoint-specific logic while the SDK manages the underlying HTTP lifecycle and Singer specification compliance.

The architecture is designed to be highly extensible, allowing developers to override specific methods for custom URL generation, complex pagination strategies, and specialized error handling. This module serves as the foundation for both standard RESTful interactions and specialized GraphQLStream implementations.

Sources: singer_sdk/streams/rest.py:34-92, singer_sdk/streams/graphql.py:23-32, CLAUDE.md:67-72

Core Architecture and Data Flow

The REST Stream architecture centers around the RESTStream and its internal _HTTPStream base class. It leverages the requests library to manage sessions and execute HTTP calls. The data flow follows a cyclic pattern: generating a request, executing it via a decorated method that handles retries, validating the response, and finally parsing records and pagination tokens.

Request Execution Flow

The following diagram illustrates the sequence of operations performed during a single sync cycle of a REST stream.

Primary Components

Component	Description
HTTPRequest	A dataclass encapsulating the URL, method, headers, parameters, and payload for a single call.
PageContext	A container for the stream's partition context and current pagination token.
RESTStream	The primary abstract base class for REST-based data extraction.
GraphQLStream	A specialized subclass for interacting with GraphQL endpoints using POST requests.

Sources: singer_sdk/streams/rest.py:64-92, singer_sdk/streams/graphql.py:23-26

Request Construction

Requests are constructed dynamically using components like url_base and path. The SDK supports path expansion where placeholders in the path string (e.g., {project_id}) are automatically replaced by values from the stream's context or configuration.

Key Methods

• get_url(context): Combines url_base and path, expanding context variables.
• get_url_params(context, next_page_token): Returns query parameters for the request.
• prepare_request_payload(context, next_page_token): Constructs the request body, typically as JSON if payload_as_json is enabled.
• get_http_request(page): Orchestrates the assembly of the HTTPRequest object.

Sources: singer_sdk/streams/rest.py:192-212, singer_sdk/streams/rest.py:393-418, packages/meltano-tap-gitlab/tap_gitlab/streams.py:61-71

Pagination Handling

The SDK provides several built-in paginators to handle different API styles. Developers should override get_new_paginator() to return the appropriate paginator instance.

Paginator Types

• JSONPathPaginator: Extracts a cursor or next-page token from the response body using a JSONPath expression.
• SimpleHeaderPaginator: Uses specific response headers (e.g., X-Next-Page) to identify the next page.
• SinglePagePaginator: Used when the endpoint does not support pagination.

Sources: singer_sdk/streams/rest.py:750-766, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:79-106, packages/meltano-tap-gitlab/tap_gitlab/streams.py:73-79

Response Validation and Error Handling

The validate_response method is responsible for checking HTTP status codes and determining if a failure is retriable or fatal.

Error Classification

• RetriableAPIError: Raised for 5xx errors, 429 (Rate Limited), and codes listed in extra_retry_statuses. These trigger the backoff logic.
• FatalAPIError: Raised for 4xx errors (excluding retriables), indicating the sync should stop.
• GraphQL Validation: GraphQLStream extends this logic to check for an errors key in the JSON response body, even if the status code is 200 OK.

Sources: singer_sdk/streams/rest.py:270-305, singer_sdk/streams/graphql.py:94-135, tests/core/rest/test_failure.py:46-105

GraphQL Specifics

The GraphQLStream class adapts the REST framework for GraphQL APIs. It defaults the http_method to POST and uses the query property to define the GraphQL query string.

Implementation Example: Gitlab Tap

Sample implementations like GitlabStream demonstrate real-world usage of these components, including custom authentication and partition-based context generation for parent-child stream relationships.

• Authentication: Uses SimpleAuthenticator with a private token header.
• Partitions: The ProjectBasedStream class generates partitions based on project_ids from the configuration to target specific URLs.
• Paginator: Uses SimpleHeaderPaginator("X-Next-Page").

Sources: packages/meltano-tap-gitlab/tap_gitlab/streams.py:38-79, packages/meltano-tap-gitlab/tap_gitlab/streams.py:82-108

REST API Streams provide a robust, standardized way to build Singer taps by abstracting the complexities of HTTP communication. By leveraging inheritance and method overriding, developers can create resilient extractors that handle everything from simple GET requests to complex paginated GraphQL APIs, while benefitting from built-in metrics and retry logic.