Secrets Management
Relevant source files
The following files were used as context for generating this wiki page:
Secrets Management in Grafana is a specialized system designed to securely store, manage, and distribute sensitive information such as database passwords, API keys, and credentials. It utilizes a resource-oriented architecture inspired by Kubernetes, allowing users to define "Keepers" (encryption providers) and "Secure Values" (the secrets themselves) through a declarative API.
The system is built upon Grafana's "Unified Storage" layer, which abstracts the underlying persistence (typically a SQL database like PostgreSQL, MySQL, or SQLite) and provides standard operations such as Create, Get, List, and Watch. This ensures that sensitive data is not stored as opaque blobs but as structured resources with associated metadata and encryption context.
Sources: devenv/secrets/README.md, contribute/architecture/k8s-inspired-backend-arch.md:46-52
Architecture and Components
The Secrets Management system is composed of several key entities that define how data is encrypted and stored. It follows the "Apps Approach" where resources are defined using CUE schemas and managed via the App SDK.
Key Entities
| Component | Description |
|---|---|
| Keeper | An encryption provider configuration. It defines how a secret is encrypted (e.g., using AWS KMS or a local secret key). |
| Secure Value | The actual sensitive data (secret). It is associated with a specific Keeper for encryption and can have associated labels and decrypters. |
| Unified Storage | The internal persistence layer that handles resource versioning and storage in the configured SQL database. |
| Secrets API | The HTTP interface located at /apis/secret.grafana.app/v1beta1 used to manage these resources. |
Sources: devenv/secrets/secrets.go:20-22, contribute/architecture/k8s-inspired-backend-arch.md:38-44, devenv/secrets/README.md
Data Flow Diagram
The following diagram illustrates how a secret (Secure Value) is created and stored within the system, utilizing a Keeper for encryption context.
Keepers and Providers
Keepers act as the administrative backbone of the secrets system. Every SecureValue must be associated with a Keeper that provides the necessary encryption keys. The system supports multiple provider types, with AWS being a primary implementation.
Keeper Configuration
A Keeper resource requires a provider-specific configuration. For AWS, this includes the region, an optional Amazon Resource Name (ARN) for role assumption, and an External ID for cross-account access.
keepers:
gdev-aws-production:
description: "Production AWS KMS"
aws:
region: "us-east-1"
assumeRoleArn: "arn:aws:iam::123:role/test"
externalID: "my-external-id"
Sources: devenv/secrets/README.md, devenv/secrets/secrets.go:49-59
System Keeper
The "system" keeper type is a built-in provider that cannot be created via the API. It is generally used for internal Grafana operations and default encryption when no external provider is specified.
Sources: devenv/secrets/README.md
Secure Values
Secure Values are the resources representing the secrets. They contain the sensitive payload and metadata defining which entities (decrypters) are authorized to access the plain-text value.
Resource Structure
Secure Values are defined with a spec containing the value and a list of authorized decrypters. They also support standard Kubernetes-style labels for organization and filtering.
{
"apiVersion": "secret.grafana.app/v1beta1",
"kind": "SecureValue",
"metadata": {
"name": "gdev-database-password",
"labels": { "env": "prod" }
},
"spec": {
"description": "Production database password",
"value": "super-secret-string",
"decrypters": ["secrets-manager-testing"]
}
}
Sources: devenv/secrets/secrets.go:81-99, devenv/secrets/README.md
API Implementation and Lifecycle
The Secrets Management API is served under the group secret.grafana.app at version v1beta1. It supports standard RESTful operations managed through the Grafana API Server.
API Endpoints
| Method | Endpoint | Description |
|---|---|---|
GET | /apis/secret.grafana.app/v1beta1/namespaces/{ns}/keepers | List all configured keepers. |
POST | /apis/secret.grafana.app/v1beta1/namespaces/{ns}/keepers | Create a new encryption keeper. |
POST | /apis/secret.grafana.app/v1beta1/namespaces/{ns}/securevalues | Create a new encrypted secret. |
DELETE | /apis/secret.grafana.app/v1beta1/namespaces/{ns}/securevalues/{name} | Delete a specific secret. |
Sources: devenv/secrets/secrets.go:111-125, devenv/secrets/README.md
Sequence: Resource Creation
The following sequence diagram represents the interaction between the client, the API, and the internal storage during the creation of a secret.
Development and Testing
For local development, Grafana provides a seeding script (devenv/secrets/secrets.go) that can be used to populate the API with test data.
Configuration for Development
Development requires specific feature toggles to be enabled in Grafana:
secretsManagementAppPlatformUIsecretsManagementAppPlatformsecretsKeeperUI
The script allows for generating bulk data for load testing using the -generate flag, which creates N keepers and N secure values with randomized but deterministic names.
Sources: devenv/secrets/README.md, devenv/secrets/secrets.go:420-445
Conclusion
The Secrets Management system provides a robust, Kubernetes-inspired framework for handling sensitive information within Grafana. By decoupling the encryption logic (Keepers) from the secret storage (Secure Values) and leveraging Unified Storage for persistence, the system achieves a high degree of consistency, auditability, and security across the platform.
Sources: contribute/architecture/k8s-inspired-backend-arch.md:1-15, devenv/secrets/README.md