Schedules & Jobs

Relevant source files

The following files were used as context for generating this wiki page:

• src/meltano/core/schedule.py
• src/meltano/schemas/meltano.schema.json
• tests/meltano/core/test_project_files.py
• integration/meltano-manifest/expected-manifests/meltano-manifest.json
• tests/fixtures/multifile_project/meltano.yml

Meltano provides a declarative framework for defining data integration workflows through Jobs and automating their recurring execution via Schedules. Jobs represent a sequence of tasks—typically involving extractors, loaders, and transformers—while Schedules bind these Jobs or ELT commands to specific time intervals using cron expressions.

Sources: src/meltano/schemas/meltano.schema.json, src/meltano/core/schedule.py:180-228

Jobs Architecture

Jobs are named collections of tasks defined within the project configuration. They allow developers to group multiple plugin executions into a single atomic unit of work.

Job Structure and Tasks

A Job is defined by a unique name and a list of tasks. Tasks can be provided as a single string or an array of strings. In complex pipelines, tasks often represent a chain of "Extractor -> Mapper -> Loader" or specific utility commands like dbt runs.

The diagram above illustrates how a Job named meltanohub_el sequences multiple plugin commands.
Sources: src/meltano/schemas/meltano.schema.json:1130-1166, integration/meltano-manifest/expected-manifests/meltano-manifest.json:44-52

Job Configuration Data Model

Field	Type	Description
name	string	Unique identifier for the job.
tasks	string or array	The list of commands or plugin sequences to execute.
annotations	object	Metadata keyed by tool/vendor name (not processed by core).

Sources: src/meltano/schemas/meltano.schema.json:1130-1160

Schedules Architecture

Schedules automate the execution of either a predefined Job or a specific ELT command (Extractor, Loader, Transform) based on a time interval.

Schedule Types

Meltano implements two primary types of schedules:

1. JobSchedule: References a named Job defined in the jobs section of the project.
2. ELTSchedule: Defines a specific ELT pipeline (extractor, loader, transform) directly within the schedule.

The class diagram shows the inheritance structure used to represent different scheduling mechanisms.
Sources: src/meltano/core/schedule.py:143-228

Cron and Interval Validation

Schedules use standard UNIX cron expressions or predefined aliases. Meltano performs strict validation on these intervals, supporting 5-field cron expressions (minute, hour, day, month, day-of-week).

Supported Cron Aliases:

Alias	Expression
@hourly	0 * * * *
@daily	0 0 * * *
@weekly	0 0 * * 0
@monthly	0 0 1 * *
@yearly	0 0 1 1 *
@midnight	0 0 * * *

Meltano also supports non-running aliases such as @once, @manual, and @none.
Sources: src/meltano/core/schedule.py:16-32, src/meltano/core/schedule.py:61-82

Schedule Execution Logic

For ELTSchedule objects, Meltano constructs a set of arguments to pass to the ELT engine, including an automatically generated state-id derived from the schedule's name to ensure incremental state persistence across runs.

The sequence shows how schedules utilize the system database to track state and history.
Sources: src/meltano/core/schedule.py:202-212, src/meltano/core/schedule.py:214-224

Configuration and Project Integration

Both Jobs and Schedules are defined in the meltano.yml file and can be distributed across multiple files using include_paths.

Manifest Representation

When Meltano compiles the project configuration into a manifest, schedules and jobs are rendered with their full environment context.

"schedules": [
  {
    "env": {},
    "interval": "0 12 * * *",
    "job": "slack_notifications",
    "name": "slack_notifications"
  }
]

Sources: integration/meltano-manifest/expected-manifests/meltano-manifest.json:1115-1121, tests/meltano/core/test_project_files.py:40-100

Validation Schema

The project schema enforces structure for these entities:

• Schedules: Must have a name, interval, and either a job reference OR an extractor, loader, and transform setting.
• Cron Pattern: Validated against the regex ^((@(hourly|daily|weekly|monthly|yearly|once))|((((\d+,)+\d+|(\d+|\*)(\/|-)\d+|\d+|\*) ?){5,6}))$.

Sources: src/meltano/schemas/meltano.schema.json:1078-1128

Summary

Schedules and Jobs form the orchestration layer of Meltano. Jobs allow for complex, multi-stage task definitions, while Schedules provide the automation logic required to run these pipelines at scale. By leveraging cron-based intervals and system database integration for state management, Meltano ensures that scheduled workflows remain reliable and observable.

Sources: src/meltano/core/schedule.py, src/meltano/schemas/meltano.schema.json