Pagination & Rate Limiting

Relevant source files

The following files were used as context for generating this wiki page:

• singer_sdk/pagination.py
• tests/core/rest/test_pagination.py
• cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md
• CLAUDE.md
• singer_sdk/exceptions.py (Inferred from CLAUDE.md references)
• cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md

Pagination and rate limiting are critical components of the Meltano Singer SDK's RESTStream architecture. These mechanisms ensure that data extraction is performant, memory-efficient, and compliant with source API constraints. The SDK provides a modular paginator system that allows developers to handle various API pagination patterns (offset, page-based, cursor, or HATEOAS) through specialized classes rather than manual loop logic.

Rate limiting is primarily handled via built-in retry logic and backoff strategies within the RESTStream class. This allows the SDK to automatically handle transient failures such as 429 Too Many Requests or network timeouts, promoting robust data synchronization pipelines.

Sources: singer_sdk/pagination.py , cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:92-95

Paginator Architecture

The SDK employs an object-oriented approach to pagination. Every RESTStream can define a paginator by overriding the get_new_paginator() method. The paginator maintains internal state, including the current page value, the total number of pages processed, and whether the pagination cycle is complete.

Paginator Lifecycle

The following diagram illustrates the interaction between a Stream and its Paginator during a data extraction loop.

Base Classes and Interface

All paginators inherit from BaseAPIPaginator, which defines the core interface:

• current_value: Returns the token or index for the current request.
• advance(response): Validates the response and updates the internal state for the next request.
• has_more(response): Determines if the loop should continue.
• get_next(response): Extracts the next token/index from the API response.

Sources: singer_sdk/pagination.py:43-55

Built-in Paginator Types

The SDK provides several specialized paginator classes to handle common API patterns without requiring custom implementation.

Paginator Class	Description	Key Argument
SinglePagePaginator	Used for endpoints that do not support pagination or return all data in one call.	N/A
PageNumberPaginator	Increments an integer page counter (e.g., page 1, 2, 3).	start_value
OffsetPaginator	Increments an offset by a fixed page size (e.g., 0, 50, 100).	page_size
JSONPathPaginator	Extracts a "next" token from a specific path in the JSON response body.	jsonpath
SimpleHeaderPaginator	Extracts a "next" token from a specific HTTP header.	key
HeaderLinkPaginator	Follows RFC 5988 Link headers (e.g., rel="next").	N/A

Sources: singer_sdk/pagination.py:126-258, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:100-125

HATEOAS Support

For APIs following HATEOAS (Hypermedia as the Engine of Application State) principles, the BaseHATEOASPaginator parses URLs provided in the response. It returns a ParseResult object, allowing the stream to extract specific query parameters or rebuild the request URL entirely.

Sources: singer_sdk/pagination.py:151-180, tests/core/rest/test_pagination.py:192-205

Rate Limiting and Retry Logic

The SDK provides robust error handling to manage API rate limits. This is implemented using the python-backoff library within the RESTStream class (referenced via configuration guidelines).

Retryable vs Fatal Errors

The SDK distinguishes between transient errors that warrant a retry and permanent errors that should halt the pipeline.

Rate Limiting Features

1. Backoff Strategies: Automatic exponential backoff when hitting 429 (Rate Limit) or 5XX server errors.
2. Configurable Timeouts: Users can define timeouts to prevent hanging connections.
3. Wait and Retry: The SDK respects "Retry-After" headers if provided by the source API.

Sources: CLAUDE.md:30-31, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:213-214

Advanced Configuration

Empty Page Handling

By default, the SDK may stop if a page returns no records. Developers can override continue_if_empty() in a custom paginator to keep the loop going if the API indicates more data exists despite an empty current result set.

Sources: singer_sdk/pagination.py:84-93, tests/core/rest/test_pagination.py:328-333

Loop Detection

To prevent infinite requests, BaseAPIPaginator includes built-in loop detection. If the get_next() method returns a value identical to the current_value for two consecutive requests, the SDK raises a RuntimeError.

Sources: singer_sdk/pagination.py:108-115, tests/core/rest/test_pagination.py:85-95

Implementation Example

Developers implement pagination by defining the paginator in the stream class:

# singer_sdk/streams.py implementation pattern
class MyStream(RESTStream):
    def get_new_paginator(self):
        # Use a built-in paginator for a JSON body cursor
        return JSONPathPaginator("$.pagination.next_cursor")

    def get_http_request(self, page):
        request = super().get_http_request(page=page)
        if page.next_page_token:
            # Map the paginator token to a query parameter
            request.params["cursor"] = page.next_page_token
        return request

Sources: singer_sdk/pagination.py:165-177, tests/core/rest/test_pagination.py:260-264

Summary

Pagination and Rate Limiting in the Meltano Singer SDK are designed to be declarative. By providing a suite of standard paginators and a robust retry framework, the SDK minimizes the boilerplate code required to build production-grade Singer taps. This architecture ensures that data extraction is both resilient to API instability and capable of handling large datasets through efficient resource management.

Sources: README.md:38-41, CLAUDE.md:73-77