Introduction to cron-ext
Relevant source files
The following files were used as context for generating this wiki page:
cron-ext is a Meltano utility extension designed to provide basic job scheduling for Meltano projects using the system's cron daemon. It functions by translating Meltano schedules into standard cron entries, allowing users to automate data integration pipelines (ELT) and arbitrary jobs without needing a dedicated external orchestrator.
Sources: README.md:3-8, pyproject.toml:4
The extension operates by creating a dedicated directory within the Meltano project to store wrapper shell scripts. These scripts capture the environment context—including the virtual environment and path variables—present during installation to ensure that cron executions mirror the manual execution environment.
Sources: README.md:14-20, cron_ext/extension.py:64-70
Core Architecture and Data Flow
The architecture of cron-ext follows a standard Meltano SDK extension pattern, wrapping core scheduling logic in a CLI accessible via meltano invoke. It interfaces with the Meltano project to retrieve schedule definitions and interacts with the system's crontab or standard output to manage schedule entries.
System Components
| Component | Responsibility |
|---|---|
Cron Extension | Main logic for processing schedules and managing files. |
EntryStore | Abstract interface for managing where cron entries are written. |
CrontabEntryStore | Implementation that interacts directly with the system crontab command. |
StdoutEntryStore | Implementation that prints generated cron entries to the console. |
Wrapper Scripts | Generated .sh files that execute meltano run or meltano elt. |
Sources: cron_ext/extension.py:30-50, cron_ext/stores/base.py, cron_ext/stores/crontab.py, cron_ext/stores/stdout.py
Installation and Execution Flow
When a user runs the install command, the extension fetches current schedules from Meltano, generates shell script wrappers, and updates the crontab.
The flow shows the process from invoking the install command to the final update of the cron storage.
Sources: cron_ext/extension.py:88-128, cron_ext/main.py:110-116
Functional Modules
Schedule Management Logic
The Cron class in extension.py is the heart of the system. It uses meltano schedule list --format=json to identify available schedules. It supports both legacy elt schedules and modern job schedules.
Key Logic:
- Environment Capture: It captures
PATHandVIRTUAL_ENVto include in the generated scripts. - Script Generation: Scripts are stored in
.meltano/run/cron-ext/and include acdto the project root and a pipe to/usr/bin/logger. - Permissions: Uses
stat.S_IXUSRto ensure the generated shell scripts are executable by the owner.
Sources: cron_ext/extension.py:64-106, README.md:14-20
Crontab Section Management
The CrontabEntryStore manages a specific section within the user's crontab delimited by comment markers containing the MELTANO_PROJECT_ID. This ensures that cron-ext only modifies entries it manages, leaving other user cron jobs untouched.
The sequence diagram illustrates how the CrontabEntryStore reads, modifies, and reapplies the crontab while maintaining project-specific sections.
Sources: cron_ext/stores/crontab.py:27-65, cron_ext/subprocess.py:24-52
CLI Commands Reference
| Command | Description | Options |
|---|---|---|
install | Installs project schedules into the target. | [schedule_ids], --target=[crontab|stdout] |
list | Lists currently installed cron entries. | --name-only, --orphaned, --target |
uninstall | Removes entries from the crontab. | [schedule_ids], --all |
describe | Shows available commands and metadata. | --format=[text|json|yaml] |
Sources: cron_ext/main.py:46-150
Internal Data Structures
Entry Parsing
The extension uses regular expressions to identify and validate cron entries belonging to the project. This is crucial for distinguishing project-managed scripts from other cron tasks.
# From cron_ext/entry.py
comment_pattern = re.compile(r"^\s*#.*$")
entry_pattern = re.compile(r"(?i)^.+?'(?P<path>/?(?:.*/)(?P<name>.*)\.sh)'.*$")
Sources: cron_ext/entry.py:5-6
The entry_pattern specifically looks for the file path of the generated shell scripts and extracts the name (schedule ID) and path for management operations like uninstallation or listing.
Sources: cron_ext/extension.py:157-164, cron_ext/entry.py:6
Conclusion
cron-ext provides a lightweight, file-based scheduling mechanism for Meltano. By generating environment-aware shell scripts and managing project-specific sections in the system crontab, it bridges the gap between manual execution and automated production schedules without requiring complex infrastructure. Its design emphasizes safety by using project identifiers to scope its modifications to the system environment.
Sources: README.md:3-30, cron_ext/extension.py:130-140