Pagination & Page Tokens
Relevant source files
The following files were used as context for generating this wiki page:
Crucible provides a standardized mechanism for handling paginated API operations through page tokens. The system ensures that long listings can be resumed across multiple requests while maintaining security and consistency through cryptographic binding.
The server runtime manages the cryptographic sealing and opening of tokens, while client SDKs provide high-level abstractions like async iterators or streams to walk through pages. Tokens are cryptographically bound to the specific service, operation, caller, and request filters to prevent token reuse or leakage across different contexts.
Sources: crates/crucible-server-runtime-rust/src/page_tokens/page_tokens.rs:18-32, crates/crucible-codegen-common/src/guide.rs:343-346
Server-Side Architecture
The PageTokens struct serves as an interceptor in the Rust server runtime. It handles the lifecycle of pagination tokens by intercepting requests, validating tokens, and binding them to the request context.
Token Security and Binding
Every token is bound to a specific request context. The system prefixes the following data before creating a digest:
- Service ID
- Operation ID
- Caller identity (issuer, subject, and tenant)
- Canonical JSON form of the request input (excluding the token and page size)
This binding ensures a token issued for one set of filters cannot be used with another.
Sources: crates/crucible-server-runtime-rust/src/page_tokens/page_tokens.rs:175-207, crates/crucible-server-runtime-rust/src/page_tokens/page_tokens.rs:242-261
Data Flow for Token Processing
The following diagram illustrates how the server processes an incoming request containing a page token:
The interceptor validates that the input is a JSON object and that the token member is a string before attempting to open the cryptographic seal.
Sources: crates/crucible-server-runtime-rust/src/page_tokens/page_tokens.rs:105-143
Configuration Options
The pagination system allows developers to configure token lifetimes and cursor limits to balance security and usability.
| Configuration | Default Value | Valid Range | Description |
|---|---|---|---|
ttl | 24 hours | 1s to 30 days | How long a token remains valid after issuance. |
max_cursor | 4096 bytes | 1 to 4096 bytes | The maximum size of the internal cursor data. |
token_length | ~5532 chars | N/A | Total character length including headers and base64 encoding. |
Sources: crates/crucible-server-runtime-rust/src/page_tokens/page_tokens.rs:10-15, crates/crucible-server-runtime-rust/src/page_tokens/page_tokens.rs:65-88
SDK Implementation
Crucible generates language-specific paginators that simplify sequential fetching. Client SDKs typically follow the output token until it is absent or empty and automatically reject repeated tokens.
Sources: crates/crucible-codegen-docs/src/markdown.rs:647-657
Implementation by Language
| Language | Paginator Mechanism | Interface Example |
|---|---|---|
| Go | Paginator Struct | paginator.NextPage(ctx) / HasMorePages() |
| TypeScript | Async Iterator | for await (const page of client.paginateOperation(input)) |
| Dart | Stream | await for (final page in client.operationPages(input)) |
| Python | Iterator / Async Iterator | for page in client.operation_pages(request) |
| Ruby | Enumerator | client.operation_pages(input).each do |page| |
Sources: crates/crucible-codegen-go/src/lib.rs:388-395, crates/crucible-codegen-ts/src/snippet.rs:271-274, crates/crucible-codegen-dart/src/snippet.rs:334-338, crates/crucible-codegen-common/src/guide.rs:343-346
Example: TypeScript Async Iterator
The TypeScript SDK uses an async generator to fetch pages. It continues as long as a response carries a next token.
// Each page is one response; the paginator sends `nextToken` for the next.
for await (const page of client.paginateListThings(input)) {
console.log(page.items);
}
Sources: crates/crucible-codegen-ts/src/snippet.rs:252-257
Model Definition
Operations are marked as paginated using the @paginated trait in the Crucible IR. This trait defines the specific members used for tokens and results.
Pagination Roles
- inputToken: The member in the request that carries the continuation token.
- outputToken: The member in the response that carries the next token.
- items: The member in the response containing the list of results.
- pageSize: Optional member in the request to limit results per page.
Sources: crates/crucible-codegen-docs/src/markdown.rs:647-653, crates/crucible-codegen-go/src/lib.rs:381-387
Crucible validates that the cryptographic tokens fit within the @length constraints defined on the token member in the model. If a model defines a token member with a maximum length of 2048, the server runtime must be configured with a matching max_cursor or it will fail during the build/prepare phase.
Sources: crates/crucible-server-runtime-rust/src/page_tokens/page_tokens.rs:163-171
Conclusion
The Pagination and Page Token system ensures that Crucible services can handle large datasets efficiently and securely. By cryptographically binding tokens to the request context and providing high-level SDK paginators, Crucible simplifies the development of robust, paginated APIs.