Schema Generation and Typing
Relevant source files
The following files were used as context for generating this wiki page:
Schema generation and typing in the Meltano Singer SDK provide a robust framework for defining, discovering, and transforming data structures. This system ensures that data extracted by taps and loaded by targets adheres to the Singer specification, facilitating seamless integration between diverse data sources and destinations.
The architecture relies on a declarative approach to typing, utilizing Python classes that map to JSON Schema types. It supports multiple methods of schema definition, including manual declaration, file-based discovery, and dynamic extraction from external metadata like OpenAPI specifications.
Architecture and Components
The schema system is built around several core components that handle the definition, retrieval, and normalization of stream structures.
Type System
The SDK provides a typing module that offers a Pythonic interface for building JSON Schema objects. Key classes include PropertiesList, Property, and various type-specific classes such as StringType, IntegerType, and ObjectType.
The diagram shows the relationship between property definitions and JSON Schema types within the SDK.
Sources: singer_sdk/typing.py, singer_sdk/helpers/capabilities.py:18-32
Schema Sources
Schema sources are responsible for retrieving schema definitions from various locations. The SchemaSource abstract base class defines the interface for these retrievals.
| Class | Description |
|---|---|
SchemaDirectory | Loads schemas from .json files within a specified directory or Python module. |
OpenAPISchema | Extracts schemas from OpenAPI (Swagger) 2.0 or 3.0 specifications, supporting both local files and remote URLs. |
StreamSchema | A Python descriptor that automatically fetches a stream's schema from a provided SchemaSource based on the stream's name. |
Sources: singer_sdk/schema/source.py:46-248, tests/core/schema/test_source.py:125-250
Schema Discovery and Retrieval
The SDK supports automated discovery of schemas, which is a critical part of the Singer discover capability.
OpenAPI Integration
OpenAPISchema allows developers to point a tap at an OpenAPI spec to automatically generate stream schemas. It handles complex tasks such as resolving $ref pointers and converting OpenAPI-specific fields to standard JSON Schema.
The flowchart illustrates the process of loading and resolving a schema from an OpenAPI specification.
Sources: singer_sdk/schema/source.py:164-248, tests/core/schema/test_source.py:270-380
Normalization Logic
During retrieval, particularly from OpenAPI, schemas undergo normalization to ensure compatibility with Singer targets. This includes:
- Nullable Handling: Converting
nullable: trueto a type array like["string", "null"]. - Keyword Stripping: Removing validation-only keywords like
enumorpatternthat may cause false negatives in different regex dialects. - Logical unwrapping: Flattening single-element
oneOforallOfconstructs.
Sources: singer_sdk/schema/source.py:250-410, tests/core/schema/test_source.py:642-835
Property Mapping and Transformation
In addition to static schemas, the SDK supports dynamic transformations via "Stream Maps." This capability allows for field renaming, computed fields, and record filtering while ensuring the schema stays in sync with the transformed data.
Key Mapping Properties
| Property | Description |
|---|---|
__alias__ | Renames a stream. |
__filter__ | A string expression used to include or exclude records. |
__source__ | Identifies the source stream for a cloned stream. |
__key_properties__ | Overrides the primary keys for a stream. |
Sources: singer_sdk/helpers/capabilities.py:48-115, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:52-75
Data Flow for Inline Mapping
When a mapper or tap uses stream maps, the record transformation process follows a specific sequence:
This diagram depicts the flow of data through the inline mapping system.
Sources: cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:20-80
Conclusion
Schema generation and typing in the Meltano Singer SDK provide a declarative and extensible way to manage data structures. By abstracting JSON Schema complexities into Python types and providing automated sources like OpenAPI, the SDK reduces the manual effort required to build compliant Singer taps and targets while maintaining strict data integrity across the ELT pipeline.
Sources: README.md, singer_sdk/typing.py, singer_sdk/schema/source.py