Quickstart Guide

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• cron_ext/extension.py
• cron_ext/main.py
• cron_ext/stores/crontab.py
• cron_ext/stores/stdout.py
• cron_ext/entry.py

The cron-ext utility is a Meltano extension designed to provide basic job scheduling by converting Meltano schedules into standard cron entries. It automates the creation of wrapper scripts that manage environment variables and virtual environments, ensuring that scheduled Meltano tasks run consistently within the project context.

Sources: README.md:3-7, cron_ext/extension.py:11-20

Installation and Configuration

To use cron-ext, it must be registered as a utility plugin within your meltano.yml file. This configuration defines the plugin's namespace, executable, and primary commands.

Plugin Registration

Add the following block to your plugins section in meltano.yml:

plugins:
  utilities:
  - name: cron
    namespace: cron
    pip_url: git+https://github.com/meltano/cron-ext.git
    executable: cron
    commands:
      describe:
        args: describe
        executable: cron

Sources: README.md:27-37

After updating the configuration, install the plugin using the Meltano CLI:

meltano add utility cron

Sources: README.md:41

Core Functionality

The extension works by reading Meltano's internal schedules (both elt and job types) and generating corresponding shell scripts and cron entries.

Execution Flow

The following diagram illustrates how a scheduled task is transformed from a Meltano definition into an active cron job:

The extension creates a managed section in the user's crontab identified by the MELTANO_PROJECT_ID. Sources: README.md:9-21, cron_ext/extension.py:73-98, cron_ext/stores/crontab.py:35-49

Component Architecture

The system uses a Strategy pattern for storing cron entries, allowing output to either the system crontab or standard output.

Usage Commands

The extension is invoked through meltano invoke cron <command>.

Common Commands

Command	Description	Relevant Arguments
install	Installs project schedules into the crontab.	schedule_ids, --target
list	Lists installed cron entries for the project.	--name-only, --orphaned, --target
uninstall	Removes managed entries from the crontab.	schedule_ids, --all, --target
describe	Shows available extension capabilities.	--format

Sources: cron_ext/main.py:65-135

Target Destinations

Users can specify where the cron entries should be sent using the --target option.

• crontab (default): Directly modifies the user's crontab file using crontab -l and crontab -.
• stdout: Prints the generated cron lines to the terminal, useful for manual verification or custom scheduling tools.

Sources: cron_ext/main.py:68, cron_ext/init.py:10-15, cron_ext/stores/stdout.py:27-28

Logic and Data Handling

Script Generation

When install is called, the extension generates a .sh file for each schedule in .meltano/run/cron-ext/. These scripts:

1. cd into the Meltano project root.
2. Export the PATH and VIRTUAL_ENV (if active).
3. Set environment variables defined in the Meltano schedule.
4. Execute meltano run or meltano elt.
5. Pipe all output (stdout and stderr) to /usr/bin/logger.

Sources: README.md:16-21, cron_ext/extension.py:84-100

Entry Management

The CrontabEntryStore identifies its managed section using specific header and footer comments containing the project ID:
# ----- BEGIN MELTANO CRONTAB SECTION (<project_id>) -----
# ----- END MELTANO CRONTAB SECTION (<project_id>) -----

Sources: cron_ext/stores/crontab.py:38-42, cron_ext/stores/crontab.py:88-95

Orphaned Schedules

A schedule is considered "orphaned" if it exists in the crontab section but is no longer defined in the current meltano.yml. The list command can filter these specifically using the --orphaned flag.

Sources: cron_ext/main.py:48-56, cron_ext/main.py:75-80

Summary

The cron-ext utility streamlines the transition from local development to scheduled production runs by handling the complexities of environment activation and logging. By managing a dedicated section within the crontab, it ensures that Meltano tasks are scheduled safely without interfering with existing system tasks.

Sources: README.md:3-14, cron_ext/extension.py:114-135