Background Services & Jobs
Relevant source files
The following files were used as context for generating this wiki page:
Background services and jobs constitute the asynchronous processing layer of the Grafana backend. These components handle long-running tasks, periodic maintenance, and state reconciliation that must occur outside the lifecycle of a standard HTTP request. By offloading these operations, Grafana ensures the API remains responsive while maintaining system health and data consistency.
The architecture relies on a structured lifecycle where services are registered via Wire dependency injection and managed by the central server. Modern implementations are increasingly moving toward a "Resource-oriented" model, utilizing reconcilers and controllers to align the current state of the system with the desired state defined in the database.
Sources: AGENTS.md, contribute/architecture/k8s-inspired-backend-arch.md
Core Architecture and Lifecycle
Grafana's background operations are primarily managed within the pkg/services/ directory. The system utilizes Wire for dependency injection to initialize these services and manage their dependencies. Services typically implement standard interfaces that allow the main server to start and stop them gracefully during the application lifecycle.
Service Registration and Initialization
Background services are initialized during the server startup phase. The backend uses a "Registry" approach and an "Apps" approach to define and start these services. In the registry approach, services are registered in pkg/server/ and initialized via wire.go.
Sources: AGENTS.md, contribute/architecture/k8s-inspired-backend-arch.md
Background Service Management
The BackgroundService interface or similar structures allow the server to execute long-running loops. For example, services like the cleanup service or alerting engines run as background processes that poll for work or wait for timed intervals.
| Component | Description |
|---|---|
| Registry | The legacy method for registering background services via Go code. |
| Apps Approach | Modern modular implementation using CUE for resource definition and controllers for logic. |
| Wire DI | Dependency injection tool used to wire service dependencies at compile time. |
Sources: contribute/architecture/k8s-inspired-backend-arch.md, AGENTS.md
Resource-Oriented background Processing
Grafana is migrating toward a Kubernetes-inspired model where background jobs are replaced by "Reconcilers." Instead of one-off scripts, these background processes continuously watch resources and ensure the actual state matches the declarative "spec."
Reconcilers and Controllers
In the "Apps" approach (found in apps/), background logic is encapsulated in reconcilers. These are background loops that:
- Watch: Monitor changes to specific resource types (e.g., Playlists, Dashboards).
- Diff: Compare the desired state in the database with the current environment.
- Act: Perform necessary updates to align the two.
The following diagram illustrates the reconciliation loop used by background services in the modern architecture:
The reconciler acts as a continuous background job, ensuring the system reaches eventual consistency.
Sources: contribute/architecture/k8s-inspired-backend-arch.md:83-110
Distributed Locking and Concurrency
To prevent background jobs from conflicting in High Availability (HA) environments, Grafana utilizes a server lock mechanism. This ensures that only one instance of a specific background job (such as a database migration or a scheduled report) runs at any given time across a cluster.
ServerLock Implementation
The serverlock package provides the infrastructure for these distributed locks. It typically uses the configured SQL database (PostgreSQL, MySQL, or SQLite) as the locking backend. This allows instances to coordinate without requiring an external system like Etcd.
This flow prevents duplicate execution of critical background tasks across multiple Grafana nodes.
Sources: pkg/infra/serverlock/serverlock.go, contribute/architecture/k8s-inspired-backend-arch.md:105-115
Alerting Background Services
Alerting is a significant consumer of background processing. The alerting engine runs as a set of background services that handle rule evaluation and notification dispatching.
Rule Evaluation
Alert rules are processed by background runners that fetch data from data sources, evaluate conditions, and update the state of the alert. This happens entirely outside the user's browser session.
Notification Policies
Once an alert is triggered, background services handle the routing through notification policies and contact points. This ensures that even if the UI is closed, notifications are processed according to the configured schedules and mute timings.
| Feature | Background Process |
|---|---|
| Evaluation | Periodic polling of data sources to check alert conditions. |
| Polling | Caching and refreshing alert rules every RULE_LIST_POLL_INTERVAL_MS. |
| Cleanup | Removing expired silences and old alert history. |
Sources: public/app/features/alerting/unified/AGENTS.md:85-110, AGENTS.md
Recommended Practices for Background Logic
When developing new background services or jobs, the following practices are recommended:
- Avoid Global State: Use dependency injection via Wire to provide services with their required configuration (
setting.Cfg) and storage handlers. - Context Propagation: Always pass
context.Contextthrough background loops to ensure they respect server shutdown signals and allow for distributed tracing. - Unified Storage: Use the Unified Storage layer for persistence to benefit from built-in
resourceVersionandWatchcapabilities. - Idempotency: Background jobs should be idempotent, especially in the reconciler model, as they may run multiple times against the same state.
Sources: contribute/backend/recommended-practices.md:65-110, contribute/architecture/k8s-inspired-backend-arch.md
Conclusion
Background services and jobs in Grafana have evolved from simple periodic Go functions to a sophisticated, resource-oriented reconciliation system. By leveraging distributed locking via serverlock and declarative logic via the App SDK, Grafana maintains a robust and scalable environment for asynchronous operations. This architecture ensures that critical tasks like alerting, data cleanup, and state synchronization remain consistent across distributed installations.