Image Replication Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Image Replication Engine is a core component of Harbor designed to synchronize container images, Helm charts, and other OCI artifacts between multiple registry instances. It operates based on user-defined policies that specify source and destination registries, resource filters, and triggering conditions. This engine facilitates high availability, load balancing, and multi-datacenter deployments by ensuring artifact consistency across distributed environments.
Sources: README.md:29-30, api/v2.0/swagger.yaml:1669-1698
Replication Architecture and Logic
The replication process is managed through a lifecycle of stages that transform resource metadata from a source registry to a destination registry. The engine leverages an adapter-based architecture, allowing Harbor to interface with various registry types (e.g., DockerHub, AWS ECR, Azure ACR) using the ArtifactRegistry interface.
Replication Execution Flow
The internal logic follows a strictly defined sequence of stages to ensure data integrity during synchronization:
- Initialization: The engine retrieves factory instances for both source and destination registry types to create functional adapters.
- Resource Fetching: The source adapter fetches artifacts that match the policy's filters (e.g., name, tag, or label filters).
- Source Assembly: Resources are populated with source registry information.
- Destination Assembly: The engine determines the final repository paths at the destination, applying namespace replacement and flattening logic.
- Push Preparation: The destination adapter prepares the target environment (e.g., creating namespaces or repositories) before the actual data transfer begins.
Sources: src/controller/replication/flow/stage.go:27-106, api/v2.0/swagger.yaml:1845-1880
The diagram above illustrates the sequential stages of the replication flow, from adapter initialization to final task execution.
Sources: src/controller/replication/flow/stage.go:27-106
Policy and Trigger Management
Replication is driven by Replication Policies, which act as blueprints for synchronization. Each policy defines "what" is replicated and "when" it happens.
Trigger Modes
The engine supports three primary triggering mechanisms:
- Manual: Executed on-demand by a user or external script.
- Scheduled: Triggered automatically based on a Cron string (e.g.,
0 0 0 * * *for daily at midnight). - Event-Based: Triggered immediately when resources are pushed or deleted locally (push-based only).
Sources: api/v2.0/swagger.yaml:1695-1710, src/portal/src/i18n/lang/en-us-lang.json:455-460
Resource Filtering
Filters allow fine-grained control over artifact synchronization. The engine supports filtering by:
- Name: Using patterns like
library/**. - Tag: Filtering specific versions (e.g.,
1.0*). - Label: Selecting artifacts based on Harbor labels.
- Resource Type: Distinguishing between images and other OCI artifacts.
Sources: src/portal/src/i18n/lang/en-us-lang.json:45-50, api/v2.0/swagger.yaml:1711-1725
Destination Namespace Mapping
A critical feature of the replication engine is the ability to map source repository structures to different destination structures. This is handled by the replaceNamespace function and the "Flattening" configuration.
Namespace Flattening Levels
The engine allows users to reduce nested repository structures when copying images. If a source repository is a/b/c/d/img and the destination namespace is ns, the following rules apply:
| Flattening Option | Resulting Path | Replace Count |
|---|---|---|
| No Flattening | ns/a/b/c/d/img | 0 |
| Flatten 1 Level | ns/b/c/d/img | 1 |
| Flatten All Levels | ns/img | -1 |
Sources: src/controller/replication/flow/stage.go:193-214, src/portal/src/i18n/lang/en-us-lang.json:488-500
This diagram shows how different flattening levels transform a complex nested source path into a destination path within the 'ns' namespace.
Sources: src/controller/replication/flow/stage.go:193-214
API Interface
The Replication Engine is exposed via a set of RESTful endpoints under the /replication path.
Key API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/replication/policies | GET | List all configured replication policies. |
/replication/policies | POST | Create a new replication policy. |
/replication/executions | POST | Start a replication execution based on a policy ID. |
/replication/executions/{id}/tasks | GET | List individual tasks (unit of work per repository) for an execution. |
/replication/adapters | GET | List supported registry adapter types. |
Sources: api/v2.0/swagger.yaml:1290-1490
Execution Statuses
Replication executions and tasks transition through various states:
- Pending: Scheduled but not yet started.
- InProgress: Currently transferring data.
- Succeed: Completed successfully.
- Failed: Encountered an error (Harbor automatically retries certain failures).
- Stopped: Manually canceled by an administrator.
Sources: api/v2.0/swagger.yaml:1810-1840, README.md:30
Conclusion
The Image Replication Engine provides a robust framework for artifact distribution. By combining an extensible adapter system with flexible namespace mapping and multi-trigger support, it ensures that container images and Helm charts are available where they are needed across hybrid and multi-cloud infrastructures. The integration of granular filters and automated retries makes it a critical component for maintaining registry consistency and operational reliability.
Sources: README.md:29-35, src/controller/replication/flow/stage.go:27-50