Background Job Service (Jobservice)
Relevant source files
The following files were used as context for generating this wiki page:
The Background Job Service (Jobservice) is a critical component of the Harbor ecosystem designed to handle and process asynchronous background tasks. Built on top of the gocraft/work job queue framework, it provides a reliable, fast, and efficient way to manage long-running tasks such as image replication, vulnerability scanning, and garbage collection.
Jobservice extends the underlying queue framework by adding a RESTful API, execution contexts, enhanced status tracking (including scheduled, stopped, and cancelled), and support for periodic jobs. It is designed for scalability and high availability, allowing multiple nodes to coordinate via a shared persistent backend (currently Redis).
Sources: src/jobservice/README.md:3-33, src/jobservice/README.md:43-52
System Architecture
The Jobservice architecture separates the control plane (Controller) from the compute nodes (Worker Pool). While typically packaged in one process, they coordinate through a virtual channel provided by the persistent data backend.
Core Components
| Component | Description |
|---|---|
| API Server | A lightweight RESTful server serving job-related requests and health checks. |
| Bootstrap | The entry point responsible for initializing context, API server, and worker pools. |
| Controller | The core coordinator that manages the overall flow of job submission and tracking. |
| Scheduler | Specifically handles the lifecycle and interval-based enqueuing of Periodic jobs. |
| Worker Pool | The execution environment where registered job implementations are run. |
| Stats Manager | Maintains real-time status, execution statistics, and manages status webhooks. |
| Persistent Driver | The backend storage layer (Redis) that persists job data and state. |
Sources: src/jobservice/README.md:35-46, src/jobservice/runtime/bootstrap.go:94-160
Architecture Overview
The following diagram illustrates the relationship between the API, Controller, and the Workers through the Redis persistence layer.
The virtual channel architecture allows jobs enqueued by a controller on one node to be picked up by a worker pool on another node, enabling horizontal scaling.
Sources: src/jobservice/README.md:35-52
Job Lifecycle and Programming Model
Jobs in Jobservice must implement a specific Go interface to be recognized by the worker pool. This model ensures that jobs provide metadata regarding retry logic and concurrency limits.
Job Interface
Every valid job must implement methods to define its behavior under failure and its resource consumption:
MaxFails(): Defines allowed retry attempts (default is 4).MaxCurrency(): Controls the maximum number of active jobs of this type within a Redis instance.ShouldRetry(): Boolean logic to determine if a failed job qualifies for another attempt.Validate(): Validates input parameters before execution.Run(): The entry point for the business logic.
Sources: src/jobservice/README.md:54-95
Job Status Transitions
Jobs transition through various states tracked by the Stats Manager. Jobservice supports more granular states than standard queue frameworks.
Job Types and Execution
Jobservice categorizes background tasks into three primary execution modes defined by their metadata.
| Job Kind | Description | Trigger |
|---|---|---|
| Generic | Standard asynchronous job executed once as soon as resources allow. | Immediate |
| Scheduled | Delayed execution intended to run at a specific future timestamp. | schedule_delay |
| Periodic | A job template that repeatedly enqueues running executions. | cron_spec |
Unique Jobs
Jobservice supports a unique flag in the metadata. When enabled, the service ensures that only one job with a specific name and argument set exists in the queue at one time, preventing redundant processing of the same task.
Sources: src/jobservice/README.md:16-24, src/jobservice/job/models.go:39-44
Sub-job Execution (Upstream Tracking)
Jobs can launch additional jobs using the LaunchJob function within their execution context. These are tracked as "executions" of the parent job. In the case of Periodic jobs, the periodic entry acts as an upstream template, while each individual run is linked via an upstream_job_id.
Sources: src/jobservice/README.md:148-164, src/jobservice/job/models.go:56-65
Configuration and Deployment
Jobservice is configured via YAML files or environment variables. It requires connection details for the Harbor Core server and the Redis backend.
Key Configuration Options
| Option | ENV Variable | Default/Description |
|---|---|---|
protocol | JOB_SERVICE_PROTOCOL | http or https |
port | JOB_SERVICE_PORT | API listening port (e.g., 9443) |
worker_pool.workers | JOB_SERVICE_POOL_WORKERS | Max concurrent jobs per process |
worker_pool.backend | JOB_SERVICE_POOL_BACKEND | Only redis is currently supported |
core_server | CORE_URL | Harbor core endpoint for config retrieval |
core_secret | CORE_SECRET | Shared secret for API authorization |
Sources: src/jobservice/README.md:204-222, src/pkg/chart/testdata/harbor-schema2/templates/jobservice/jobservice-dpl.yaml:94-118
Security and Authorization
Jobservice API requests must include an Authorization header using the Harbor-Secret scheme. This secret is provided to the service at runtime via the CORE_SECRET environment variable.
// Authorization header requirement
Authorization: Harbor-Secret <secret>
Sources: src/jobservice/README.md:258-265, src/jobservice/runtime/bootstrap.go:217-219
API Endpoints Summary
Jobservice provides a specialized internal API for job management.
| Method | Path | Description |
|---|---|---|
POST | /api/v1/jobs | Submit a new job (Generic, Scheduled, or Periodic). |
GET | /api/v1/jobs/{job_id} | Get detailed stats and execution info for a job. |
POST | /api/v1/jobs/{job_id} | Perform actions: stop, cancel, or retry. |
GET | /api/v1/jobs/{job_id}/log | Retrieve execution logs for a specific job. |
GET | /api/v1/stats | Retrieve worker pool health and global stats. |
Sources: src/jobservice/README.md:267-375, api/v2.0/swagger.yaml:1210-1310
Conclusion
The Harbor Jobservice acts as the engine for all non-blocking operations within the registry. By leveraging a Redis-backed queue and providing a robust Go programming model for job implementers, it ensures that critical tasks like data synchronization and system maintenance are performed reliably and scalably. Its integration with the Harbor Core via shared secrets and centralized logging provides administrators with full visibility into the background state of the registry.
Sources: src/jobservice/README.md:1-15, src/jobservice/runtime/bootstrap.go:70-90