Project Scaffolding Templates

Relevant source files

The following files were used as context for generating this wiki page:

• cookiecutter/tap-template/cookiecutter.json
• cookiecutter/mapper-template/cookiecutter.json
• cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md
• cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md
• cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md
• cookiecutter/tap-template/{{cookiecutter.tap_id}}/meltano.yml
• cookiecutter/tap-template/{{cookiecutter.tap_id}}/pyproject.toml

The Project Scaffolding Templates are a collection of cookiecutter configurations designed to accelerate the development of Singer-compliant taps, targets, and mappers. These templates provide a standardized starting point, ensuring that new plugins adhere to the Meltano Singer SDK's architectural patterns and best practices.

The scaffolding system automates the creation of boilerplate code, directory structures, and configuration files such as pyproject.toml and meltano.yml. It supports various stream types (REST, GraphQL, SQL) and authentication methods, allowing developers to focus on the specific logic of the data source or destination rather than fundamental Singer specification requirements.

Sources: README.md, cookiecutter/tap-template/cookiecutter.json:1-40, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:1-15

Template Architecture and Flow

The scaffolding process follows a standard generator pattern where user-provided variables are used to populate Jinja2-style placeholders within the template files.

The diagram above illustrates the workflow of using the scaffolding templates to generate a standardized project structure.
Sources: cookiecutter/tap-template/cookiecutter.json, cookiecutter/mapper-template/cookiecutter.json

Supported Plugin Types

The SDK provides three primary scaffolding templates, each tailored to a specific component of the Singer ecosystem.

Tap Scaffolding

Used for building data extractors. It provides boilerplate for API clients, stream definitions, and schema management.

• Key Components: tap.py (entry point), client.py (API handler), and streams.py (data definitions).
• Capabilities: Includes support for incremental sync, pagination (Offset, PageNumber, Cursor), and authentication.

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:10-50, cookiecutter/tap-template/cookiecutter.json:8-18

Target Scaffolding

Used for building data loaders. It handles the ingestion of Singer records and state.

• Serialization Methods: Supports "Per record", "Per batch", and "SQL" (SQLAlchemy-based) loading strategies.
• Key Components: target.py (configuration) and sinks.py (loading logic).

Sources: cookiecutter/target-template/{{cookiecutter.target_id}}/AGENTS.md:10-40

Mapper Scaffolding

Used for in-flight data transformations.

• Capabilities: Provides templates for stream aliasing, field renaming, computed fields, and record filtering.
• Key Components: mapper.py utilizing the InlineMapper class.

Sources: cookiecutter/mapper-template/{{cookiecutter.mapper_id}}/AGENTS.md:10-30

Configuration and Metadata

Scaffolded projects include pre-configured metadata files that integrate with the Python ecosystem and Meltano.

Configuration Parameters

The following table describes common variables used across scaffolding templates:

Variable	Description	Example
source_name / name	CamelCase identifier for Python classes	MySource
tap_id / mapper_id	kebab-case identifier for the CLI tool	tap-mysource
library_name	snake_case identifier for Python imports	tap_mysource
auth_method	Selected authentication strategy	OAuth2, API Key, JWT
stream_type	Transport layer protocol	REST, GraphQL, SQL

Sources: cookiecutter/tap-template/cookiecutter.json:1-35, cookiecutter/mapper-template/cookiecutter.json:1-30

Integrated Tooling Configuration

The templates generate a pyproject.toml file configured for modern Python development using uv and hatch.

[project]
name = "{{cookiecutter.tap_id}}"
dynamic = ["version"]
dependencies = [
    "singer-sdk~=0.54.5",
    "requests~=2.34.0", # Included if REST/GraphQL selected
]

[project.scripts]
{{cookiecutter.tap_id}} = '{{cookiecutter.library_name}}.tap:Tap{{cookiecutter.source_name}}.cli'

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/pyproject.toml:20-60

Singer-SDK Integration

The scaffolding reinforces the architectural hierarchy of the Meltano Singer SDK.

The diagram shows the inheritance structure provided in the tap scaffolding for different API types.
Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:50-80

Development Environment Setup

Each scaffolded project includes standard documentation and automation files:

1. README.md: Standard installation and usage instructions.
2. AGENTS.md: Specific instructions for AI coding agents regarding SDK patterns like pagination and state.
3. meltano.yml: A project file for immediate testing with the Meltano ecosystem, including predefined extractors or loaders settings.

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/README.md:50-100, cookiecutter/tap-template/{{cookiecutter.tap_id}}/meltano.yml:1-30, CLAUDE.md:1-20

Summary

Project Scaffolding Templates provide a robust, production-ready foundation for Singer connectors. By automating the setup of dependencies, CLI entry points, and SDK class structures, the templates ensure that developers can quickly implement data integration logic while maintaining compatibility with the Singer specification and the Meltano orchestration platform.