Data Flattening & Type Conformance
Relevant source files
The following files were used as context for generating this wiki page:
Data Flattening and Type Conformance are critical processes within the Meltano Singer SDK that ensure data extracted from varied sources is transformed into a structure compatible with downstream targets. Flattening specifically refers to the conversion of nested JSON objects into a flat, tabular-like structure by concatenating parent and child keys, while type conformance ensures that the data adheres to the expected schema types defined in the Singer specification.
These features are primarily utilized by Mappers to transform data in-flight and by Targets to prepare records for destinations that do not support nested structures, such as relational databases or CSV files. The SDK provides internal helpers to automate these transformations based on user configuration, including depth limits and key naming conventions.
Sources: singer_sdk/helpers/_flattening.py, cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:30-45
Flattening Architecture and Logic
The flattening logic in the SDK is designed to recursively traverse JSON schemas and records, collapsing nested objects into a single level. This is controlled by a set of configuration options that define how deep the recursion should go and how the resulting flat keys should be formatted.
Flattening Configuration
The FlatteningOptions dataclass encapsulates the rules for the flattening process. It includes settings for enabling the feature, the maximum recursion depth, key separators, and maximum key lengths to prevent issues with destination systems that have identifier limits.
| Option | Type | Default | Description |
|---|---|---|---|
enabled | bool | Required | Whether flattening is active. |
max_level | int | -1 | The max recursion level. Zero-based and exclusive. |
separator | str | __ | String used to concatenate nested key names. |
max_key_length | int | 255 | The maximum allowable length for a flattened key. |
Sources: singer_sdk/helpers/_flattening.py:16-52
Key Flattening Process
When a nested object is encountered, the SDK generates a new key by joining the parent key path with the current key name using the configured separator. If the resulting key exceeds the max_key_length, the SDK applies an inflection-based reduction algorithm (removing lowercase letters from camelized versions of the keys) to shorten the identifier while maintaining uniqueness.
The diagram shows the logic flow for generating flattened keys and the recursive reduction of key length if limits are exceeded.
Schema and Record Transformation
The SDK provides separate functions for flattening schemas and the records that follow them. This ensures that the metadata (SCHEMA messages) and the data (RECORD messages) remain in sync during a Singer stream.
Schema Flattening
The flatten_schema function takes a standard JSON schema and returns a flattened version. Objects are collapsed into strings or other primitive types depending on the max_level.
- Depth Control: If the
levelis less thanmax_level, recursion continues intoproperties. - Fallback Types: If an object or array is reached at the
max_level, it is typically typed as astring(often containing serialized JSON) to ensure compatibility. - Composite Types: The logic handles
oneOfandanyOfby identifying the primary type (e.g., string, array, or object) and marking it as nullable.
Sources: singer_sdk/helpers/_flattening.py:103-315
Record Flattening
The flatten_record function transforms actual data dictionaries. It uses the flattened_schema as a reference to determine if a value should be recursively flattened or serialized as a JSON string.
The sequence illustrates how the record flattening logic decides whether to dive deeper into a nested object or terminate and serialize the value.
Type Conformance and Serialization
Type conformance ensures that values emitted by a tap or modified by a mapper match the destination's expectations. The SDK uses serialize_json for complex types (dictionaries and lists) that cannot be flattened further.
Serialization Rules
The function _should_jsondump_value determines if a value requires JSON serialization.
- Implicit Collections: All
dictandlistinstances are serialized. - Schema-driven: If the flattened schema defines a field as a null-object-array type (
{"null", "object", "array"}), the value is serialized to a string.
Sources: singer_sdk/helpers/_flattening.py:396-417, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:120-135
Mapping and Transformation
Inline Mappers utilize these utilities to perform complex transformations. Mappers can:
- Rename fields: Mapping nested paths like
customer.nametocustomer_name. - Hash/Mask data: Transforming sensitive fields (e.g.,
md5(email)) during the flattening process. - Type Casting: Using Python-like expressions to convert types (e.g.,
str(numeric_field)).
Sources: cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:40-65, cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:270-285
Technical Implementation Summary
| Component | File Path | Responsibility |
|---|---|---|
FlatteningOptions | singer_sdk/helpers/_flattening.py | Validates and stores flattening config. |
flatten_key | singer_sdk/helpers/_flattening.py | Handles key concatenation and length reduction. |
flatten_schema | singer_sdk/helpers/_flattening.py | Recursively transforms JSON schemas. |
flatten_record | singer_sdk/helpers/_flattening.py | Recursively transforms data records. |
SQLSink | cookiecutter/target-template/target/AGENTS.md | Uses flattened data for automatic table creation. |
Sources: singer_sdk/helpers/_flattening.py, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:70-85, pyproject.toml:50-70
Data Flattening and Type Conformance provide a robust mechanism for bridge-building between highly nested semi-structured data sources (like REST APIs) and flat destination systems. By automating key generation, depth control, and type serialization, the SDK reduces the manual overhead required to build Singer-compliant connectors.