Generalized Metadata Service (GMS)
Relevant source files
The following files were used as context for generating this wiki page:
The Generalized Metadata Service (GMS) serves as the core backend service layer for DataHub. It provides a unified platform for storing, retrieving, and managing metadata across the entire data ecosystem. GMS acts as the primary authority for the Metadata Graph, exposing technical metadata, business context, and lineage through a standardized set of APIs including GraphQL, Rest.li, and OpenAPI.
Sources: docs-website/static/llms.txt:3-7, datahub-graphql-core/README.md:6-10
As a central component, GMS facilitates the "Metadata Serving" architecture, enabling both human users (via the DataHub UI) and machine agents (via SDKs and the Model Context Protocol) to interact with the metadata. It supports an extensible metadata model consisting of entities (e.g., Datasets, CorpUsers) and aspects (e.g., Ownership, Status).
Sources: docs-website/static/llms.txt:3-7, docs-website/sidebars.js:833-839
Architecture and Components
GMS is structured to support a high-scale, event-driven metadata architecture. It sits between the metadata ingestion framework and the presentation layers (Web UI and APIs).
Metadata Interaction Flow
The following diagram illustrates how GMS interacts with other major components of the DataHub ecosystem to serve and store metadata.
Core GMS Components
GMS functionality is realized through several key sub-systems and libraries:
| Component | Description |
|---|---|
| Entity Service | Manages the lifecycle of metadata entities and aspects, including CRUD operations and versioning. |
| Search Service | Handles indexing and searching across metadata entities using specialized indices. |
| Rest.li API | The primary internal and legacy external API for fine-grained metadata operations. |
| GraphQL Engine | A shared library (datahub-graphql-core) that provides a graph-based representation on top of GMS. |
Sources: datahub-graphql-core/README.md:12-25, metadata-service/restli-api/src/main/snapshot/com.linkedin.operations.operations.snapshot.json:1150-1160
API Surface and Maintenance
GMS exposes administrative and maintenance endpoints, primarily through the Rest.li operations resource. These endpoints allow operators to perform system-level tasks such as index management and data truncation.
Rest.li Operations API
The operations resource provides specialized actions for managing the state of GMS indices and time-series data.
| Action | Parameters | Description |
|---|---|---|
restoreIndices | aspect, urn, batchSize, limit | Re-indexes metadata from primary storage into search/graph indices. |
truncateTimeseriesAspect | entityType, aspect, endTimeMillis | Deletes historical time-series metadata older than a specified timestamp. |
getIndexSizes | N/A | Returns the storage size in MB for various metadata indices. |
getEsTaskStatus | nodeId, taskId | Checks the status of long-running Elasticsearch tasks triggered by GMS. |
GraphQL Integration
The datahub-graphql-core module provides a GraphQL facade over the GMS service layer. This allows for efficient, batched data fetching and complex relationship traversal.
Sources: datahub-graphql-core/README.md:12-20
Extending the GMS Graph
Developers can extend the GMS graph by following a standardized process:
- GMS Schema Definition: Define types and inputs in
.graphqlfiles based on GMS Pegasus models. - Entity Resolvers: Implement
EntityTypeclasses to define how to fetch the entity from GMS. - Data Mappers: Implement mappers to transform GMS Pegasus models into GraphQL-friendly POJOs.
- Wiring: Register the new types and resolvers in
GmsGraphQLEngine.
Sources: datahub-graphql-core/README.md:33-85
Conclusion
The Generalized Metadata Service (GMS) is the foundational "context platform" of DataHub. By providing a unified API for a complex metadata graph, it enables cross-functional use cases like data discovery, lineage visualization, and automated governance. Its design emphasizes extensibility and integration, supporting a wide range of clients from standard web browsers to advanced AI agents via the Model Context Protocol.
Sources: docs-website/static/llms.txt:3-12, datahub-graphql-core/README.md:12-25