SQL Database & Migrations

Relevant source files

The following files were used as context for generating this wiki page:

• apps/dashboard/pkg/migration/schemaversion/README.md
• pkg/storage/unified/migrations/README.md
• apps/dashboard/pkg/migration/README.md
• contribute/developer-guide.md
• contribute/backend/recommended-practices.md
• AGENTS.md

Grafana utilizes a relational SQL database to store core resources such as dashboards, folders, playlists, and data sources. The system employs a robust migration framework to ensure data integrity and backward compatibility as the software evolves. This system is divided into two primary categories: legacy SQL migrations and the newer Unified Storage migrations.

The migration system handles everything from low-level database schema changes (SQL tables) to high-level resource transformations (Dashboard JSON schema versions). It is designed to run automatically during Grafana startup, ensuring that the database state matches the requirements of the running binary.

Architecture and Migration Flow

The migration architecture is designed to handle the transition of resources from legacy SQL storage to a unified storage backend. This process is managed by a MigrationRunner that executes per-organization logic to stream resources.

Dashboard Schema Migrations

Dashboard schema migrations are specialized transformations that ensure older dashboard JSON models remain compatible with newer Grafana versions. These migrations are sequential and versioned.

Versioning and Logic

Dashboards use a schemaVersion property to track their evolution. The system currently supports versions from v13 up to v42. Each version has a corresponding Go function that transforms the dashboard map.

Component	Description
Minimum Version	v13
Latest Version	v42
Backend Location	apps/dashboard/pkg/migration/schemaversion/
Registration	Migrations must be registered in migrations.go

Sources: apps/dashboard/pkg/migration/schemaversion/README.md

Implementation Example

A typical schema migration function updates the version and modifies specific properties:

func V42(ctx context.Context, dashboard map[string]interface{}) error {
	dashboard["schemaVersion"] = 42
	// Transformation logic here
	return nil
}

Sources: apps/dashboard/pkg/migration/schemaversion/README.md

Unified Storage Migrations

Unified storage migrations facilitate the movement of resources from legacy SQL tables to the new unified backend. This system supports folders, dashboards, playlists, short URLs, and data sources.

Key Components

Component	Responsibility
MigrationRegistry	Thread-safe registry of all resource migration definitions.
UnifiedMigrator	Interface for streaming legacy resources to the new backend.
Validators	Ensures data integrity (e.g., CountValidator, FolderTreeValidator).
TableLocker	Prevents concurrent writes by locking legacy tables during migration.
TableRenamer	Appends _legacy to SQL tables post-migration to prevent stale writes.

Sources: pkg/storage/unified/migrations/README.md

Table Renaming Strategy

To prevent data inconsistency during rolling upgrades, legacy tables are renamed. The mechanism varies by database:

• Postgres: Uses LOCK TABLE ... IN SHARE MODE and ALTER TABLE ... RENAME TO.
• MySQL: Uses LOCK TABLES ... READ and separate connections for non-transactional DDL.
• SQLite: Uses a shared transaction for the rename operation.

Sources: pkg/storage/unified/migrations/README.md

Observability and Monitoring

The migration system provides extensive instrumentation via Prometheus metrics and structured logging to track conversion success and data loss.

Migration Metrics

The following metrics are tracked during API and schema conversions:

• grafana_dashboard_migration_conversion_success_total: Counter for successful migrations.
• grafana_dashboard_migration_conversion_failure_total: Counter for failures, labeled with error_type.

Error Classification

Errors are classified into specific types to aid debugging:

1. conversion_error: General API-level issues.
2. schema_version_migration_error: Failures in specific version functions (e.g., v15 -> v16).
3. schema_minimum_version_error: Dashboard version is below the supported minimum.
4. conversion_data_loss_error: Detected decrease in panels, queries, or annotations.

Sources: apps/dashboard/pkg/migration/README.md

Recommended Practices for SQL

The Grafana project maintains several recommended practices for database interactions to ensure maintainability and cross-database compatibility:

• Avoid Global State: Migrating away from global xorm engine variables to instance-based SqlStore handlers.
• Epoch-based Dates: Newly introduced date columns should be stored as Unix timestamps (integers) to simplify cross-database date comparisons.
• Context Propagation: The context.Context must be propagated through all layers, including the database layer, to support tracing and request cancellation.
• No init(): Avoid using init() for database setup; use dependency injection via Wire instead.

Sources: contribute/backend/recommended-practices.md, AGENTS.md

Development and Testing

Developers can interact with the SQL environment and migrations through specific tools:

Database Environment Setup

To run integration tests against specific SQL backends, use the provided developer environment:

# Start PostgreSQL and MySQL containers
make devenv sources=mysql_tests,postgres_tests

# Run integration tests for MySQL
make test-go-integration-mysql

Sources: contribute/developer-guide.md

Re-running Migrations

To force a migration to run again (e.g., for debugging), the corresponding entry must be deleted from the status table:

DELETE FROM unifiedstorage_migration_log WHERE migration_id = 'folders and dashboards migration';

Sources: pkg/storage/unified/migrations/README.md

Data Loss Detection

The system automatically compares resource counts (panels, queries, annotations, links, variables) before and after migration. Any decrease triggers a conversion_data_loss_error and is recorded in the logs.
Sources: apps/dashboard/pkg/migration/README.md