Telemetry & Tracking
Relevant source files
The following files were used as context for generating this wiki page:
- CHANGELOG.md
- README.md
- integration/meltano-manifest/expected-manifests/meltano-manifest.json
- integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json
- integration/meltano-manifest/expected-manifests/meltano-manifest.cicd.json
- integration/meltano-manifest/expected-manifests/meltano-manifest.userdev.json
Telemetry and Tracking in Meltano is a declarative system used to monitor project performance, collect anonymous usage statistics, and capture diagnostic data. The system is designed to provide visibility into how the platform is used while respecting user privacy through configurable opt-out mechanisms.
The tracking infrastructure relies on Snowplow collectors and is primarily managed through project-level configurations and environment variables. Recent updates have focused on ensuring that tracking honor explicit user "disable" settings and streamlining how metrics are rendered within the Meltano CLI environment.
Sources: CHANGELOG.md, README.md
Core Configuration & Environment
The telemetry behavior is defined within the Meltano project manifest and controlled via specific environment variables. Each Meltano project is assigned a unique project_id to distinguish tracking events across different environments.
Telemetry Settings
The following environment variables govern the telemetry subsystem:
| Setting | Type | Default (Typical) | Description |
|---|---|---|---|
MELTANO_SEND_ANONYMOUS_USAGE_STATS | Boolean | false | Global toggle for sending anonymous usage data. |
MELTANO_SNOWPLOW_COLLECTOR_ENDPOINTS | List[URL] | ["https://sp.meltano.com"] | Target URLs for the Snowplow telemetry collector. |
MELTANO_PROJECT_ID | UUID | Generated | Unique identifier for the specific project instance. |
MELTANO_CLI_LOG_LEVEL | String | info | Controls the verbosity of telemetry-related logs. |
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.json:28-31, integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json:30-31
Environment-Specific Tracking
Tracking configurations often vary between development and production environments. For instance, production manifests typically define standard Snowplow endpoints, while CI/CD or user development manifests may override these values or disable tracking entirely to avoid polluting production metrics.
Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.cicd.json, integration/meltano-manifest/expected-manifests/meltano-manifest.userdev.json
This diagram illustrates the logic flow when a Meltano command is executed and its relationship to the telemetry subsystem.
Metrics & Event Capture
Meltano captures specific events related to the lifecycle of data integration tasks. This includes plugin execution, job status, and CLI interactions.
Singer SDK Logging & Metrics
The platform supports parsing structured logs from the Singer SDK. Recent enhancements have simplified how these metrics are rendered in the console, moving toward a more concise output format while maintaining the ability to capture detailed error exceptions from structured logs.
Sources: CHANGELOG.md:162, CHANGELOG.md:174
Telemetry Logic Refinements
Key logic changes documented in recent versions include:
- Tracking Honesty: A critical fix was implemented to ensure the system strictly honors the
disable_trackingsetting (Issue #9456). - Metric Simplification: How metrics are rendered with the console logger has been simplified for better readability (Issue #9583).
- Subprocess rename: Subprocess loggers were renamed to
meltano.plugins.<stream>.<type>.<name>to better separate stdout and stderr streams for more accurate tracking of plugin behavior.
Sources: CHANGELOG.md:161, CHANGELOG.md:195, CHANGELOG.md:255
Data Flow Architecture
Telemetry data flows from individual plugin executions and CLI commands through the Meltano core and out to remote collectors.
The sequence diagram shows the interaction between the CLI, the Core tracking logic, and the external Snowplow collector.
Logging Integration
Telemetry is closely tied to the logging subsystem. The MELTANO_CLI_LOG_CONFIG typically points to a logging.yaml file that defines how tracking events are surfaced to the user. Meltano now uses a standard platform-dependent location to store these logs, ensuring consistency across operating systems.
Sources: CHANGELOG.md:162, integration/meltano-manifest/expected-manifests/meltano-manifest.json:20
Key Components for Log Parsing
- Log Parsers: Meltano includes specific log parsers for the Singer SDK to extract metrics from plugin output streams.
- Log Format: The default format is
colored, though telemetry events are typically processed as structured JSON before being sent to collectors.
Sources: CHANGELOG.md:174, integration/meltano-manifest/expected-manifests/meltano-manifest.json:21