Jobs & Background Workers

Relevant source files

The following files were used as context for generating this wiki page:

• tldw_Server_API/app/core/Jobs/README.md
• AGENTS.md
• Docs/Architecture.md
• Docs/Product/Watchlists/Watchlist_PRD.md
• Docs/Code_Documentation/Embeddings-Documentation.md
• tldw_Server_API/app/core/Watchlists/README.md

The Jobs and Background Workers system in tldw_server provides a framework for durable, asynchronous task execution. It is designed to handle long-running operations such as audio processing, embedding generation, prompt optimization, and media ingestion. The system supports multiple backends, including SQLite and PostgreSQL, and incorporates features like task leasing, retries, quarantine for failed tasks, and per-owner quotas.

The architecture distinguishes between Jobs and the Scheduler. Jobs are intended for user-facing work requiring administrative controls (pause/resume/drain), while the Scheduler handles internal system orchestration with complex task dependencies.

Sources: tldw_Server_API/app/core/Jobs/README.md, AGENTS.md:124-142, Docs/Architecture.md:41-45

Core Architecture and Data Flow

The background work system is built around a centralized JobManager that orchestrates the lifecycle of a task from creation to completion or failure.

Job Lifecycle and Leasing

When a job is created, it is enqueued into a domain-specific queue. Workers interact with the JobManager to "lease" jobs, ensuring that a single task is not processed by multiple workers simultaneously. If a job fails, the system manages retries or moves the job to a "quarantine" state if retry limits are exceeded.

Sources: tldw_Server_API/app/core/Jobs/README.md, Docs/Code_Documentation/Embeddings-Documentation.md

Jobs vs. Scheduler Decision Guide

The project employs two distinct background execution paths based on the nature of the task.

Feature	Use Jobs	Use Scheduler
Primary Purpose	User-facing/Ops visibility	Internal system orchestration
Control Plane	Pause, Resume, Drain, Retries	Task dependencies, Idempotency keys
Visibility	Admin API endpoints & Quotas	Handler registration (@task)
Worker Model	Independent WorkerSDK processes	Core Scheduler worker pool
Example Use Case	Reading Digest, Audio Transcoding	Watchlist monitors, Workflow steps

Sources: AGENTS.md:124-142

Component Breakdown

JobManager

The JobManager is the primary entry point for managing the lifecycle of background work. It handles:

• Job Creation: Enqueuing work for domains such as audio, embeddings, and connectors.
• Queue Control: Implementing administrative actions like pausing or draining specific queues.
• Maintenance: Functions for rescheduling stuck jobs, immediate retries, and pruning old job records.
• Metrics: Exporting job counts and statuses for dashboarding.

Sources: tldw_Server_API/app/core/Jobs/README.md

Workers and SDK

Workers are implemented using the worker_sdk.py. These processes are responsible for executing the business logic associated with a job type.

• Domain Scoping: Workers are typically scoped to a specific domain (e.g., embeddings) or job type.
• Leasing: Workers lease jobs to prevent concurrency issues.
• Payloads: Job payloads must be JSON-serializable to ensure compatibility with the database backends.

Sources: tldw_Server_API/app/core/Jobs/README.md, Docs/Code_Documentation/Embeddings-Documentation.md

Domain-Specific Implementations

Embeddings Worker Architecture

The embeddings system utilizes a Redis Streams worker architecture for multi-tenant deployments. Jobs serve as the root record for status and billing in this context.

Watchlist Pipeline

Watchlists utilize the scheduler.py (APScheduler) to trigger periodic monitors. These monitors then enqueue jobs for fetching and filtering content.

• Flow: Scheduler → Watchlist Pipeline → Ingestion Jobs.
• Monitoring: Supports live status updates via WebSockets for active runs.

Sources: tldw_Server_API/app/core/Watchlists/README.md, Docs/Product/Watchlists/Watchlist_PRD.md:255-265

Administrative and Operational Controls

The system provides several endpoints for managing background work. These are generally restricted to users with the admin role.

Generic Jobs Admin Endpoints

Endpoint	Method	Description
/api/v1/jobs/prune	POST	Removes old job records based on age or status.
/api/v1/jobs/queue/control	POST	Changes queue states (Pause/Resume/Drain).
/api/v1/jobs/queue/status	GET	Returns the current state of queues.
/api/v1/jobs/reschedule	POST	Manually moves stuck jobs back to the pending state.
/api/v1/jobs/retry-now	POST	Forces immediate execution of a job.

Sources: tldw_Server_API/app/core/Jobs/README.md

Configuration Options

The jobs system behavior is governed by several environment variables and configuration settings.

Variable	Default/Type	Description
JOBS_DB_URL	String	If prefixed with postgres, enables PostgreSQL backend; otherwise SQLite.
JOBS_REQUIRE_CONFIRM	Boolean	Requires X-Confirm: true header for destructive prune operations.
JOBS_UPDATE_GAUGES_ON_PRUNE	Boolean	Refreshes Prometheus metrics immediately after a prune.
FILES_JOBS_WORKER_ENABLED	Boolean	Enables the in-process worker for file artifacts exports.

Sources: tldw_Server_API/app/core/Jobs/README.md

Security and Multi-Tenancy

Jobs are secured via the project's standard AuthNZ (Authentication and Authorization) layer.

• Role-Based Access: Administrative endpoints require require_roles("admin").
• Row-Level Security (RLS): When using the PostgreSQL backend, RLS is applied per-user and per-domain before any row mutations occur.
• Quotas: The JobManager enforces per-owner quotas to prevent resource exhaustion in multi-user environments.

Sources: tldw_Server_API/app/core/Jobs/README.md, Docs/Architecture.md:213-224

Summary

Jobs and Background Workers in tldw_server provide the infrastructure for reliable, scalable media processing. By separating user-facing work (Jobs) from internal orchestration (Scheduler), the system allows for fine-grained administrative control while maintaining high-performance background execution. The integration with multiple backends and standard AuthNZ ensures the system is suitable for both local single-user and large-scale multi-tenant deployments.