Relevant source files

The following files were used as context for generating this wiki page: - [app/vmselect/main.go](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/main/app/vmselect/main.go) - [app/vminsert/main.go](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/main/app/vminsert/main.go) - [lib/storage/storage.go](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/main/lib/storage/storage.go) - [lib/tenantmetrics/counter_map.go](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/main/lib/tenantmetrics/counter_map.go) - [app/vmagent/main.go](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/main/app/vmagent/main.go) - [apptest/tests/multitenant_test.go](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/main/apptest/tests/multitenant_test.go) - [app/vmctl/native/client.go](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/main/app/vmctl/native/client.go)

# Multi-tenancy VictoriaMetrics provides built-in support for multi-tenancy, allowing multiple isolated users (tenants) to share the same infrastructure. Each tenant's data is logically separated using a combination of an `accountID` and a `projectID`. This isolation applies to both data ingestion (insert) and data retrieval (querying) paths. The system is designed to handle per-tenant metrics, resource limits, and specific API routing. Multi-tenancy is particularly relevant for the Cluster version of VictoriaMetrics, where components like `vminsert` and `vmselect` route requests based on tenant identifiers provided in the URL or via specific labels. Sources: [apptest/tests/multitenant_test.go:1-30](), [README.md:40-60]() ## Data Model and Identification A tenant is uniquely identified by a `TenantID` structure consisting of two unsigned 32-bit integers. Sources: [lib/tenantmetrics/counter_map.go:13-17]() | Field | Type | Description | | :--- | :--- | :--- | | `AccountID` | `uint32` | The primary identifier for the tenant account. | | `ProjectID` | `uint32` | A sub-identifier for projects within an account. | ### Multitenant Labeling When multi-tenancy is active, VictoriaMetrics can automatically inject or filter based on internal labels that represent the tenant. - `vm_account_id`: Represents the Account ID. - `vm_project_id`: Represents the Project ID. Sources: [apptest/tests/multitenant_test.go:68-75]() ### Tenant ID Representation The system often represents a tenant in strings as `AccountID:ProjectID` (e.g., `1:15`). If only one ID is provided, the other defaults to `0`. Sources: [apptest/tests/multitenant_test.go:48-55](), [lib/tenantmetrics/counter_map.go:81-90]() ## Architecture and Data Flow The multi-tenancy logic is distributed across different components to ensure that data is correctly routed and isolated. ### Ingestion Flow In `vmagent` and `vminsert`, multi-tenancy is enabled through specific path parsing. The system extracts the tenant information from the URL path before processing the payload. Sources: [app/vmagent/main.go:153-165]() ```mermaid graph TD Client[Client Request] --> Router{Multitenancy Enabled?} Router -- Yes --> PathParser[Parse Path for TenantID] PathParser --> AuthToken[Generate Auth Token] AuthToken --> Handler[Protocol Specific Handler] Router -- No --> Handler Handler --> Storage[Storage Ingestion] ``` The diagram shows how incoming requests are intercepted to extract tenant information before being passed to protocol handlers. Sources: [app/vmagent/main.go:153-170](), [app/vmagent/main.go:394-405]() ### Query and Selection Flow `vmselect` uses the tenant information to filter the underlying storage. It can also perform "multitenant" queries that aggregate data across all tenants if a special `multitenant` keyword is used in the query options. Sources: [apptest/tests/multitenant_test.go:62-75]() ## API Implementation Multi-tenancy impacts the URL structure for Prometheus-compatible endpoints. ### Tenant-Specific Endpoints For standard operations, the tenant is included in the URL: - Ingestion: `/insert/:/prometheus/api/v1/write` - Query: `/select/:/prometheus/api/v1/query` Sources: [app/vmagent/main.go:400-410](), [app/vmctl/native/client.go:39-42]() ### Global Multitenant Discovery The system provides admin endpoints to discover existing tenants within the storage. Sources: [app/vmctl/native/client.go:21-25]() | Endpoint | Method | Description | | :--- | :--- | :--- | | `admin/tenants` | GET | Returns a list of all active tenant IDs. | | `api/v1/label/__name__/values` | GET | Can be scoped to a specific tenant via `/select//...` | ## Internal Monitoring and Metrics VictoriaMetrics tracks performance and usage on a per-tenant basis using a `CounterMap`. This structure maintains a map of counters keyed by `TenantID`. Sources: [lib/tenantmetrics/counter_map.go:19-24]() ### Metric Name Generation Metrics for specific tenants are formatted to include the IDs as labels internally, even if they are stored in a unified map. - Standard tenant: `metric_name{accountID="1",projectID="15"}` - Multitenant aggregate: `metric_name{accountID="multitenant",projectID="multitenant"}` Sources: [lib/tenantmetrics/counter_map.go:72-90]() ```mermaid sequenceDiagram participant App as "Application Logic" participant CM as "CounterMap" participant M as "metrics.Counter" App->>CM: Get(auth.Token) CM->>CM: Convert Token to TenantID alt Tenant exists in map CM-->>App: Return existing Counter else New Tenant CM->>CM: createMetricName(ID) CM->>M: GetOrCreateCounter(name) CM-->>App: Return new Counter end ``` This sequence shows the lazy initialization of per-tenant metrics within the `CounterMap`. Sources: [lib/tenantmetrics/counter_map.go:50-70]() ## Multi-tenancy in vmagent In `vmagent`, multi-tenancy allows the agent to act as a proxy for multiple tenants simultaneously. It supports various protocols including: - InfluxDB Line Protocol - DataDog - OpenTelemetry - NewRelic - Zabbix When `MultitenancyEnabled()` returns true, `vmagent` parses the `/insert//...` prefix to determine the destination tenant for the incoming data. Sources: [app/vmagent/main.go:394-410](), [app/vmagent/main.go:420-430]() ## Summary Multi-tenancy in VictoriaMetrics is a core architectural feature that provides logical data separation through `accountID` and `projectID`. It is implemented via URL path routing, internal label injection (`vm_account_id`), and per-tenant resource monitoring. The system supports both isolated tenant access and global administrative views across all tenants. Sources: [apptest/tests/multitenant_test.go:48-120](), [lib/tenantmetrics/counter_map.go:1-20]()