Target Fundamentals
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Target Fundamentals encompass the core architectural patterns and implementation requirements for building data loaders (Targets) using the Meltano Singer SDK. Targets are responsible for consuming data produced by Taps in the Singer format and persisting it to a specific destination, such as a database, an API, or a file system.
The SDK simplifies Target development by providing abstract base classes that handle the Singer specification's complexities, including message parsing, state management, and schema validation. By utilizing these fundamentals, developers can reduce boilerplate code by approximately 70% while ensuring compliance with the Singer Spec.
Sources: README.md:5-9, README.md:43-46, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:12-14
Core Architecture
Targets built with the SDK typically consist of three primary components that define how data is received, processed, and written to the destination.
Component Hierarchy
- Target Class: The main entry point for the application. It defines the configuration schema (
config_jsonschema) and manages the initialization of sinks. - Sink Class: Handles the actual data loading and transformation logic for specific streams.
- Connector: (Specifically for SQL targets) Manages the connection pool and execution of statements against a database.
The following diagram illustrates the relationship between these classes:
The SDK uses abstract base classes to provide a plugin system where Target is the main entry point and Sink, BatchSink, or SQLSink handle data loading.
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:16-20, CLAUDE.md:129-136
Serialization Methods
The SDK supports multiple serialization patterns to accommodate different destination types and performance requirements.
Comparison of Loading Strategies
| Method | Implementation Class | Description | Use Case |
|---|---|---|---|
| Per Record | Sink | Processes each record individually via process_record(). | REST APIs, Webhooks, Message Queues. |
| Per Batch | BatchSink | Accumulates records in memory until a max_size is reached, then calls process_batch(). | Bulk APIs, File uploads. |
| SQL | SQLSink | Inherits database-specific logic like schema evolution, MERGE/UPSERT, and type mapping. | PostgreSQL, MySQL, SQLite, Snowflake. |
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:28-76
Per-Batch Data Flow
When using batch serialization, the Target follows a specific lifecycle to manage memory and maximize throughput.
Records are accumulated up to a max_size (default 10,000) before being processed as a group to improve throughput.
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:46-55
Configuration and Metadata
Targets rely on a JSON Schema-based configuration system. Settings must be defined in the Target class to ensure proper validation and discovery.
Setting Mappings
When using a Target within Meltano, the Python types defined in the SDK must match the Meltano "kind" in meltano.yml.
| Python Type (singer_sdk.typing) | Meltano Kind |
|---|---|
StringType | string |
IntegerType | integer |
BooleanType | boolean |
DateTimeType | date_iso8601 |
ArrayType | array |
ObjectType | object |
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:209-217
State Management
Targets are responsible for receiving and forwarding state messages. These messages act as checkpoints, allowing Taps to resume incremental syncs from the last successful point.
- Checkpointing: Targets should emit state messages only after the associated data has been successfully committed to the destination.
- Relaying: Targets must not modify state messages; they should be emitted as received to ensure the Tap's bookmark remains accurate.
Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:179-188, singer_sdk/testing/target_tests.py:90-99
Testing Fundamentals
The SDK includes standard test suites to verify Target compliance with the Singer specification. These tests cover edge cases such as:
- Schema Evolution: Handling schema updates during a sync.
- Data Integrity: Handling special characters, encoded strings, and complex nested objects.
- Validation: Ensuring the target fails appropriately when receiving records without a schema or records missing primary keys.
Sources: singer_sdk/testing/target_tests.py:20-165, CLAUDE.md:143-145
Standard Test Execution
Developers can use pytest with the SDK's built-in templates to run a suite of automated checks.
# singer_sdk/testing/target_tests.py:44-51
class TargetCliPrintsTest(TargetTestTemplate):
"""Test Target correctly prints version and about information."""
name = "cli_prints"
@override
def test(self) -> None:
"""Run test."""
self.target.print_version()
self.target.print_about()
Exception Handling
Robust targets must distinguish between different failure modes to allow for appropriate recovery strategies. The SDK provides a hierarchy of exceptions:
| Exception Category | Strategy |
|---|---|
FatalAPIError | Abort sync immediately (e.g., Auth failure). |
RetriableAPIError | Safe to retry (e.g., Network timeout). |
ConfigValidationError | Configuration is invalid; requires user correction. |
RecordsWithoutSchemaException | Data arrived before a schema was defined. |
Sources: CLAUDE.md:76-88, singer_sdk/testing/target_tests.py:126-129
Conclusion
Target Fundamentals provide the structural components necessary for efficient, spec-compliant data loading. By leveraging the SDK's Target and Sink abstractions, developers can implement sophisticated features like batching, SQL schema management, and automated state checkpointing with minimal effort. This architectural consistency ensures that Singer targets remain interoperable across the Meltano ecosystem.
Sources: README.md:43-52, cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:244-249