Logging and Metrics Collection

Relevant source files

The following files were used as context for generating this wiki page:

• singer_sdk/logging.py
• singer_sdk/metrics.py
• singer_sdk/_logging.py
• singer_sdk/schemas/logs.schema.json
• cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md
• CLAUDE.md

Logging and Metrics Collection in the Meltano Singer SDK provides a structured framework for observing the internal state, performance, and operational health of Singer taps, targets, and mappers. The system is designed to provide developers with actionable diagnostic information while emitting standardized metrics that can be used to monitor data throughput and latency.

The architecture emphasizes structured logging, ensuring that logs are not merely strings but rich objects containing metadata such as process IDs, logger names, and stream-specific context. This data is critical for debugging complex ELT pipelines and providing visibility into the progress of data synchronization.

Sources: singer_sdk/schemas/logs.schema.json, CLAUDE.md, cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md

Structured Logging Architecture

The SDK utilizes a structured logging approach where log events adhere to a predefined JSON schema. This allows for consistent parsing by downstream log aggregators and Meltano's internal monitoring tools.

Log Event Structure

Every log event produced by the SDK includes a mandatory set of fields that define the context of the message. These fields include information about the environment, the specific application (tap/target), and the stream currently being processed.

Field	Type	Description
level	string	The log level (e.g., debug, info, warning, error, critical).
pid	integer	The process ID of the running plugin.
logger_name	string	Name of the logger (e.g., singer_sdk.metrics).
ts	number	Timestamp of the log emission.
app_name	string	Name of the application (e.g., tap-github).
message	string	The human-readable log message.

Sources: singer_sdk/schemas/logs.schema.json:7-142

Exception Handling in Logs

When an error occurs, the logging system captures a detailed exception object. This object includes the exception type, module, message, and a full traceback containing filenames, function names, and line numbers. It also supports nested exceptions via the cause and context fields to represent Python's exception chaining.

Sources: singer_sdk/schemas/logs.schema.json:67-127, CLAUDE.md:44-59

Metrics Collection

Metrics are a specialized form of structured logging used to track quantitative data about the execution of a Singer plugin. They are emitted with a specific metric_info object in the log payload.

Metric Types and Tags

The SDK primarily tracks two types of metrics:

1. Counters: Used for accumulating values, such as record_count.
2. Timers: Used for measuring duration, such as http_request_duration.

Metrics are enriched with tags, which are key-value pairs providing additional context, such as the stream_name or endpoint being accessed.

Sources: singer_sdk/schemas/logs.schema.json:44-66, singer_sdk/metrics.py

Metrics Data Model

Metric Field	Type	Description
type	string	The category of metric (counter or timer).
metric	string	The name of the specific metric.
value	number	The numeric value recorded.
tags	object	Additional metadata for filtering metrics.

Sources: singer_sdk/schemas/logs.schema.json:48-64

Implementation for Developers

Developers building taps and targets use the self.logger attribute provided by the base classes to emit logs. The SDK handles the population of structured fields like app_name and pid automatically.

Logging Best Practices

• Structured Logging: Use self.logger to ensure logs are captured in the SDK's standard format.
• Level Usage: Follow standard levels (INFO for progress, ERROR for fatal issues, DEBUG for developer-specific traces).
• Stream Context: Logs should ideally include the stream_name when processing specific data partitions.

Sources: cookiecutter/tap-template/{{cookiecutter.tap_id}}/AGENTS.md:214-222, CLAUDE.md:86-90

Metric Emission Flow

Metrics are typically emitted during data extraction (in Taps) or data loading (in Targets). For example, a Tap might emit a metric for every batch of records fetched from a REST API.

Error Categories and Logging

The SDK defines a hierarchy of exceptions that dictate how errors are logged and whether the sync process should continue.

Exception Base	Recovery Strategy	Logging Behavior
FatalAPIError	Abort Sync	Logged as CRITICAL with full exception block.
RetriableAPIError	Retry	Logged as WARNING/INFO with retry attempt details.
IgnorableAPIError	Skip Silently	Logged as DEBUG or ignored.
ConfigValidationError	Abort Sync	Logged as ERROR highlighting specific config fields.

Sources: CLAUDE.md:44-59

Conclusion

The Logging and Metrics Collection system in the Meltano Singer SDK transforms standard text-based output into a machine-readable stream of events. By adhering to the logs.schema.json, the SDK ensures that all taps and targets provide high-quality observability data, enabling robust monitoring of ELT pipelines through record counts, execution timers, and detailed error tracebacks.