Background Jobs Processing
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Background Jobs Processing system within the TAMV Online project is designed to handle asynchronous tasks, long-running processes, and scheduled maintenance operations. By offloading these tasks from the main application thread, the system ensures high availability and responsiveness for end-user interactions. This module acts as the backbone for automated data synchronization, notification dispatching, and heavy computational logic.
The system utilizes a distributed architecture involving producers (application services), a message broker (Redis/RabbitMQ), and consumer workers that execute the actual business logic. This decoupling allows for independent scaling of the worker pool based on the volume of the task queue.
Sources: README.md:1-2, jobs/worker.py:1-5, config/queue_config.yaml:1-10
Architecture and Components
The architecture follows a standard Producer-Consumer pattern. Tasks are serialized and pushed into specific queues based on priority and type. A central scheduler handles time-based triggers, while worker nodes listen for incoming messages to process.
Workflow Logic
The following diagram illustrates the lifecycle of a background job from initiation to completion:
The flow demonstrates how both application services and the internal scheduler feed tasks into the broker, where workers then pick them up for execution.
Sources: jobs/worker.py:10-45, jobs/scheduler.py:5-20
Key Components
| Component | Responsibility | Relevant Files |
|---|---|---|
| Worker | Consumes tasks from the queue and executes defined Python functions. | jobs/worker.py |
| Scheduler | Manages CRON-like tasks and periodic maintenance jobs. | jobs/scheduler.py |
| Broker | Acts as the message intermediary (configured via YAML). | config/queue_config.yaml |
| Persistence | Stores the state and history of all jobs for auditing. | models/job_schema.sql |
Sources: jobs/worker.py, jobs/scheduler.py, config/queue_config.yaml, models/job_schema.sql
Data Model and Configuration
The system relies on a structured SQL schema to track the state of every background task. This allows for monitoring, manual intervention, and historical analysis of system performance.
Job Schema Definition
The jobs table tracks the metadata and execution metrics:
| Field | Type | Description |
|---|---|---|
id | UUID | Primary identifier for the job. |
task_name | String | The identifier for the function to execute. |
status | Enum | Current state: QUEUED, RUNNING, COMPLETED, FAILED. |
payload | JSON | Arguments passed to the task. |
retries | Integer | Number of attempts made. |
created_at | Timestamp | When the job was first enqueued. |
Sources: models/job_schema.sql:1-15
Configuration Parameters
The system behavior is governed by a central configuration file that defines queue priorities and connection strings.
queues:
high_priority:
concurrency: 5
timeout: 30s
default:
concurrency: 2
timeout: 60s
broker:
url: "redis://localhost:6379/0"
retry_on_failure: true
Sources: config/queue_config.yaml:1-12
Job Execution Sequence
When a worker picks up a task, it follows a strict sequence to ensure data integrity and error handling.
This sequence ensures that a job is never lost; if a worker crashes before completion, the message remains in the broker or is marked as failed in the database.
Sources: jobs/worker.py:50-85, models/job_schema.sql:10-20
Conclusion
The Background Jobs Processing system provides a robust framework for managing asynchronous operations within TAMV Online. By utilizing a combination of scheduled triggers and worker-based execution, it ensures that the system can handle high-load scenarios without impacting user experience. The integrated monitoring through the SQL schema and YAML configuration allows for fine-tuned control over the processing environment.
Sources: README.md:1-2, jobs/worker.py:100-110