Evaluation Job Queue System
Relevant source files
The following files were used as context for generating this wiki page:
The Evaluation Job Queue System manages the lifecycle of asynchronous evaluation tasks within the platform. It enables the distribution of heavy computational workloads, such as model inference and metric calculation, across multiple worker nodes to maintain application responsiveness.
The system utilizes a distributed task queue architecture where the API backend acts as a producer and Celery workers act as consumers. Redis serves as the message broker, facilitating communication and task persistence between components.
Sources: README.md, backend/app/core/celery_app.py, docker-compose.yml
System Architecture
The architecture consists of four primary components: the REST API, the Redis Message Broker, the Celery Worker, and the PostgreSQL Database. This decoupled design allows the platform to scale workers independently of the web server.
The diagram shows the flow from job submission to execution and data persistence.
Sources: docker-compose.yml, backend/app/worker/tasks.py
Job Lifecycle and States
The system tracks every evaluation through a state machine defined in the database model. This ensures observability and allows users to monitor progress through the API.
| State | Description |
|---|---|
PENDING | Job is created in the database and waiting to be picked up by a worker. |
STARTED | A worker has received the task and initiated the evaluation logic. |
SUCCESS | The evaluation completed without errors and results are stored. |
FAILURE | An exception occurred during processing; error details are captured. |
RETRY | The task encountered a transient error and is scheduled for another attempt. |
Sources: backend/app/models/job.py, backend/app/worker/tasks.py
Task Submission Process
When you submit an evaluation request, the system performs the following actions:
- The API validates the input parameters (model ID, dataset ID, metrics).
- The backend creates a new entry in the
jobstable with aPENDINGstatus. - The backend dispatches a Celery task with the unique Job ID to the Redis broker.
- The API returns the Job ID to the client immediately.
Sources: backend/app/api/v1/endpoints/jobs.py
Worker Implementation
Workers are dedicated processes that execute the actual evaluation code. They remain decoupled from the web environment to prevent long-running tasks from blocking HTTP requests.
The sequence diagram illustrates the asynchronous communication between the producer and the consumer via the broker.
Sources: backend/app/worker/tasks.py, backend/app/core/celery_app.py
Configuration
The system uses environment variables to configure the connection between the backend and the queue.
CELERY_BROKER_URL: Defines the connection string for Redis (e.g.,redis://redis:6379/0).CELERY_RESULT_BACKEND: Specifies where task results are temporarily cached.WORKER_CONCURRENCY: Determines the number of simultaneous tasks a single worker process can handle.
Sources: backend/app/core/celery_app.py, docker-compose.yml
Error Handling and Retries
The queue system manages execution failures through built-in retry logic. If a connection to a remote model provider fails, the worker catches the exception and schedules a retry based on a backoff policy. If the maximum number of retries is exceeded, the job status is set to FAILURE in the database, and the error stack trace is recorded in the error_log field.
Sources: backend/app/worker/tasks.py:45-60, backend/app/models/job.py
Summary
The Evaluation Job Queue System provides a robust foundation for handling long-running AI evaluation tasks. By combining FastAPI, Celery, and Redis, the platform achieves reliable task execution, clear state management, and horizontal scalability. This architecture ensures that even complex evaluations involving large datasets do not degrade the user experience of the primary interface.