Metrics & Observability
Relevant source files
The following files were used as context for generating this wiki page:
Metrics and observability in Harbor Next are centered around a dedicated exporter component that integrates with the Prometheus ecosystem. This system provides real-time visibility into the health, performance, and operational status of Harbor components, including the core services, database, and background job processing engine. The observability stack allows administrators to monitor resource consumption, system health, and complex workflows like image replication and vulnerability scanning.
The infrastructure relies on the Prometheus client library to collect data via various collectors and expose it through a standardized HTTP endpoint. This enables external monitoring systems to scrape metrics such as total storage consumption, project counts, and job execution statistics.
Sources: src/cmd/exporter/main.go, src/jobservice/README.md
Exporter Architecture
The Harbor Exporter is a standalone service that acts as a bridge between internal Harbor data and external monitoring tools. It implements the Prometheus collector interface to gather metrics across multiple domains.
Component Overview
The exporter initializes a HTTP server that listens for scrape requests. It registers various collectors, including system collectors for high-level statistics and specific collectors for components like the Job Service.
The diagram above illustrates how the Exporter's System Collector retrieves raw data from the Harbor database and registers it with the Prometheus registry to be served over HTTP.
Sources: src/cmd/exporter/main.go:34-60, src/pkg/exporter/exporter.go:25-45
Key Functions and Structures
| Element | Type | Description |
|---|---|---|
main() | Function | Entry point that parses configuration and starts the HTTP server. |
systemCollector | Struct | Implements prometheus.Collector to gather Harbor system-wide metrics. |
Describe() | Method | Sends descriptors of metrics to the Prometheus channel. |
Collect() | Method | Fetches values from Harbor and sends them as Prometheus metrics. |
Sources: src/cmd/exporter/main.go:34, src/pkg/exporter/system_collector.go:30-40
System Metrics Collection
The system collector specifically targets global Harbor statistics. It interacts with the persistent storage layer to provide insights into project distribution and storage utilization.
Collected Metrics
The following metrics are defined within the system collector:
- Total Project Count: Tracks the number of projects, categorized by public or private access.
- Total Repository Count: Monitors the number of image repositories across the entire registry.
- Storage Consumption: Reports the total size of blobs stored in the registry in bytes.
This sequence shows the synchronous flow of data from the database to the Prometheus scraper during a metrics request.
Sources: src/pkg/exporter/system_collector.go:48-90, api/v2.0/swagger.yaml:definitions/Statistic
Implementation Detail: System Collector
The Collect method in system_collector.go uses a "must-fetch" pattern where it attempts to retrieve the Statistic data model. If successful, it updates the Prometheus gauges with the retrieved values.
func (c *systemCollector) Collect(ch chan<- prometheus.Metric) {
stats, err := c.fetchStats()
if err != nil {
return
}
ch <- prometheus.MustNewConstMetric(c.totalProjectDesc, prometheus.GaugeValue, float64(stats.TotalProjectCount))
ch <- prometheus.MustNewConstMetric(c.totalRepoDesc, prometheus.GaugeValue, float64(stats.TotalRepoCount))
ch <- prometheus.MustNewConstMetric(c.storageDesc, prometheus.GaugeValue, float64(stats.TotalStorageConsumption))
}
Sources: src/pkg/exporter/system_collector.go:70-85
Job Service Observability
The Job Service is a critical component for asynchronous tasks like garbage collection, replication, and scanning. It provides its own set of observability features, including health checks and job status monitoring.
Job Status and Stats
The Job Service tracks the state of background workers and job queues. This information is exposed via specific API endpoints for administrative consumption.
| Endpoint | Method | Purpose |
|---|---|---|
/api/v1/stats | GET | Returns a list of worker pools, their concurrency, and health status. |
/api/v1/jobs/{job_id} | GET | Provides detailed status, including check-in messages and execution progress. |
/api/v1/jobs/{job_id}/log | GET | Retrieves the execution log for a specific job instance. |
Sources: src/jobservice/README.md:API, api/v2.0/swagger.yaml:paths/jobservice/pools
Job Execution Context
Jobs in Harbor Next provide observability through "Check-in" messages. This allows long-running tasks to report granular progress (e.g., "30% complete") back to the monitoring system.
Sources: src/jobservice/README.md:Check In Message
The diagram shows how the Job Service programming model enables observability within the business logic of a task.
Sources: src/jobservice/README.md:Architecture
Health Monitoring
Harbor components provide standardized health check endpoints to ensure high availability and facilitate automated recovery by orchestrators like Kubernetes.
Overall Health Status
The /health endpoint provides an aggregated status of all dependent Harbor components.
| Component | Monitored Attribute |
|---|---|
| Core | API Server availability |
| Database | Connectivity and schema status |
| Redis | Job queue backend accessibility |
| Job Service | Worker pool health and heartbeat |
Sources: api/v2.0/swagger.yaml:definitions/OverallHealthStatus, api/v2.0/swagger.yaml:paths/health
Data Model: Health Status
OverallHealthStatus:
type: object
properties:
status:
type: string
description: "healthy" only when all components are "healthy"
components:
type: array
items:
$ref: '#/definitions/ComponentHealthStatus'
Sources: api/v2.0/swagger.yaml:definitions/OverallHealthStatus
Conclusion
Metrics and Observability in Harbor Next provide a robust framework for monitoring registry health and performance. By combining a Prometheus-compatible Exporter for system-wide metrics with granular job-level tracking in the Job Service, the system ensures that administrators have the visibility required to maintain a production-grade OCI artifact registry. The integration of health endpoints further supports automated infrastructure management and reliability.