Entity Registry & Metadata Modeling
Relevant source files
The following files were used as context for generating this wiki page:
The Entity Registry and Metadata Modeling framework in DataHub provides a "no-code" or "low-code" approach to defining data entities and their associated metadata (aspects). This system allows developers to extend the core DataHub model by defining new types, relationships, and validation logic without necessarily forking the main repository.
The architecture is centered around a unified context graph where technical metadata and business context are connected via URNs (Unique Resource Names). The modeling framework handles everything from schema definition in PDL (Pegasus Data Language) to automated documentation generation and frontend rendering.
Sources: metadata-models-custom/README.md, datahub-graphql-core/README.md
Core Components and Structure
Entities and Aspects
The metadata model is built on two primary primitives:
- Entities: The primary nodes in the metadata graph (e.g., Dataset, Dashboard, CorpUser). Each entity is identified by a unique URN.
- Aspects: Atomic units of metadata attached to an entity. Aspects are versioned and represent specific categories of information such as
Ownership,GlobalTags, orDatasetProperties.
Sources: metadata-models-custom/README.md:10-20, datahub-ingestion/scripts/modeldocgen.py:73-85
The Entity Registry Configuration
The entity-registry.yml file is the central configuration for defining how entities and aspects are composed. It specifies:
- Registry ID: A unique identifier for the model registry.
- Entity Definitions: A mapping of entity names to their supported aspects.
- Key Aspect: The specific aspect that defines the identity/URN structure of the entity.
The diagram above illustrates the relationship between a Registry, the Entities it defines, and the Aspects that compose those entities.
Sources: metadata-models-custom/README.md:33-50, metadata-ingestion/scripts/modeldocgen.py:73-100
Custom Plugin Ecosystem
DataHub provides a plugin framework to control metadata lifecycle operations. These plugins allow for validation, mutation, and side effects during metadata ingestion or updates.
Plugin Types
| Plugin Type | Purpose | Execution Point |
|---|---|---|
| Validators | Ensure aspects adhere to rules (e.g., range checks, null checks). | On Request / Pre-Commit |
| Mutators | Transform data before/after persistence (e.g., lowercasing strings). | Read / Write |
| MCP Side Effects | Generate new aspects based on an input Metadata Change Proposal. | Persistence Layer |
| MCL Side Effects | Generate side effects based on a Metadata Change Log (e.g., timeseries events). | Search Index Layer |
Sources: metadata-models-custom/README.md:150-180
Metadata Processing Flow
The following sequence shows how a proposed metadata change is processed through the registry's plugin ecosystem:
Technical Modeling and Tooling
Automated Documentation Generation
The modeldocgen.py script automates the creation of technical reference guides by parsing Avro schemas and entity registries. It extracts:
- Searchable Fields: Fields annotated with
@Searchable. - Relationships: Fields annotated with
@Relationship, defining edges in the graph. - Lineage: Automatic lineage extraction from schemas.
Sources: metadata-ingestion/scripts/modeldocgen.py:150-250, metadata-ingestion/scripts/modeldocgen.py:650-700
OpenAPI Integration
OpenAPI schemas for entities are dynamically generated. The system uses specific suffixes to distinguish between request and response models:
_v2: The versioned model suffix.Request_v2: Used for write operations.Response_v2: Used for read operations.AspectRequest_v2/AspectResponse_v2: Wrappers for individual aspects with system metadata.
Sources: buildSrc/src/main/java/io/datahubproject/OpenApiEntities.java:31-45
Frontend Integration: The Entity Registry
In the React web application, the EntityRegistry acts as a layer of abstraction for rendering different entity types. It decouples the UI from the underlying GMS models.
Key Functional Responsibilities
- Profiles: Rendering the detailed view of an entity instance.
- Previews: Rendering small detail cards for search results or browse views.
- Mapping: Transforming Pegasus (GMS) models to GraphQL POJOs for the UI.
Summary
Entity Registry & Metadata Modeling provides a robust framework for managing the DataHub knowledge graph. By combining configuration-driven entity definitions with a powerful plugin ecosystem (Validators, Mutators, and Side Effects), DataHub ensures that metadata remains high-quality, interoperable, and extensible. The tight integration between backend schema definitions, automated documentation tools like modeldocgen.py, and the frontend EntityRegistry allows for a seamless experience when extending the data platform.