Extending DataHub (Models & Plugins)
Relevant source files
The following files were used as context for generating this wiki page:
DataHub provides a flexible framework for extending its core metadata model and functionality without requiring deep changes to the primary codebase. Users can define custom aspects, introduce new entities, and implement plugins to validate, mutate, or trigger side effects based on metadata changes. This extensibility ensures that DataHub can be tailored to specific organizational needs, such as custom data quality rules or specialized governance workflows.
The extension ecosystem covers the backend metadata service (GMS), the GraphQL API layer, and the React frontend. By leveraging the Metadata Model and the Custom Plugin Ecosystem, developers can enforce business rules and automate metadata enrichment through a standardized lifecycle.
Sources: metadata-models-custom/README.md:1-20, contrib/metadata-model-extensions/README.md:1-10
Metadata Model Extensions
Extending the metadata model involves defining new aspects or entities using the Pegasus Data Layout (PDL) language. These definitions are packaged as plugins and loaded into the Generalized Metadata Service (GMS).
Custom Aspects and Entities
Custom aspects are defined in PDL files located in src/main/pegasus. Each file name must match the class name, and the directory path must match the package path. Once defined, these aspects must be registered in an entity-registry.yaml file to be attached to specific entities.
The diagram above shows the lifecycle of creating and deploying a custom metadata aspect in DataHub.
Deployment and Validation
Custom models are deployed to ~/.datahub/plugins/models/ by default. The status of loaded models can be verified via the /config endpoint. DataHub performs backward compatibility checks during the loading process to prevent system corruption.
Sources: metadata-models-custom/README.md:73-88, metadata-models-custom/README.md:144-149
| Component | Description |
|---|---|
| PDL File | The schema definition for the metadata aspect. |
| entity-registry.yaml | Maps aspects to entities and defines plugin configurations. |
| AspectRetriever | Interface used by plugins to look up other aspects for validation or mutation. |
Sources: metadata-models-custom/README.md:22-60, metadata-models-custom/README.md:200-205
Custom Plugin Ecosystem
DataHub's plugin framework allows advanced users to control how metadata instances are validated, mutated, and how they generate side effects.
Sources: metadata-models-custom/README.md:155-170
Architecture Overview
This flow represents the progression of a metadata aspect through validation, mutation, and side-effect generation during a write operation.
Plugin Types
| Plugin Type | Integration Point | Purpose |
|---|---|---|
| Validators | on request or pre-commit | Enforce business rules (e.g., null checks, range checks). |
| Mutators | write or read | Modify data before storage or after retrieval (e.g., lowercasing strings). |
| MCP Side Effects | Post-persistence | Generate additional aspects in primary storage based on input. |
| MCL Side Effects | Post-persistence | Generate search-only aspects or timeseries events. |
Sources: metadata-models-custom/README.md:195-360
Spring Support
Plugins can utilize Spring for dependency injection. By setting spring.enabled: true in the registry and providing a packageScan path, DataHub isolates the plugin class loader to avoid conflicts with core system classes.
Sources: metadata-models-custom/README.md:405-430
GraphQL and Frontend Extensions
Extending the backend often requires corresponding changes to the API and UI layers to surface the new metadata.
GraphQL API Core
To add an entity to the GraphQL graph, developers must:
- Extend
entity.graphqlwith new types or inputs. - Implement
EntityTypeclasses (e.g.,DatasetType.java) to map GMS models to GraphQL POJOs. - Register the new type in
GmsGraphQLEngine.java.
Sources: datahub-graphql-core/README.md:32-65
React UI Extensions
The DataHub React app is designed for configurability and extensibility. Key steps for adding a new entity to the UI include:
- GraphQL Queries: Declare queries in
.graphqlfiles to fetch the new entity data. - EntityRegistry: Register the new entity instance in
App.tsxso it can be managed by the registry layer. - Profiles & Previews: Implement profile views and smaller detail cards for the entity.
Sources: datahub-web-react/README.md:22-45, datahub-web-react/README.md:188-215
Metadata Ingestion Source Extensions
Adding a new ingestion source requires extending the Python ingestion framework.
Development Workflow
- Config Model: Inherit from
ConfigModelusing Pydantic for configuration validation. - Implementation: Override
get_workunits_internalto produce a stream of Metadata Change Proposals (MCPs). - Discoverability: Declare the source under
entry_pointsinsetup.pyto allow the CLI to recognize the new plugin.
Sources: metadata-ingestion/adding-source.md:30-65
Documentation and UI Integration
New sources should use decorators like @platform_name and @capability for automatic documentation generation. To display the source in the UI's Managed Ingestion tab, it must be added to sources.json and constants.ts in the datahub-web-react module.
Sources: metadata-ingestion/adding-source.md:78-115, metadata-ingestion/adding-source.md:215-235
Conclusion
Extending DataHub is a multi-layered process involving model definition, plugin-based logic, and frontend registration. Whether through custom PDL aspects, Java-based validators, or Python ingestion connectors, these extension points allow DataHub to evolve into a specialized metadata platform while maintaining core stability.
Sources: metadata-models-custom/README.md:450-460, contrib/metadata-model-extensions/README.md:55-70