DataHub Actions & Automations
Relevant source files
The following files were used as context for generating this wiki page:
DataHub Actions & Automations constitute an event-driven framework designed to enable real-time responses to metadata changes within the DataHub ecosystem. This system allows users to automate workflows such as documentation propagation, tag synchronization, and integration with external communication tools like Slack or Microsoft Teams. By acting on metadata changes in real-time, organizations can maintain a high degree of data governance and observability without manual intervention.
The framework supports a wide range of active metadata workflows, including propagating Glossary Terms and Documentation automatically across related entities. It also facilitates integration with DataHub Cloud features like AI-powered documentation generation and automated anomaly detection through assertions.
Sources: docs-website/static/llms.txt:68-71, docs-website/sidebars.js:235-261
Framework Architecture
The DataHub Actions framework is structured into three primary components: Sources, Events, and Actions. These components work together in a pipeline to detect metadata changes and execute specific logic in response.
Core Components
| Component | Description |
|---|---|
| Sources | The origin of the event stream, typically monitoring DataHub's Metadata Change Log (MCL). |
| Events | Specific triggers within the metadata stream, such as entity creation, tag updates, or schema changes. |
| Actions | The logic executed when an event occurs, such as sending a notification or updating another metadata aspect. |
Sources: docs-website/sidebars.js:685-715, docs-website/static/llms.txt:115
Data Flow for Automations
The following diagram illustrates the flow from a metadata change in a source system to the execution of an automated action in DataHub.
The diagram shows how metadata changes flow through the GMS (Generalized Metadata Service) to trigger the Actions framework.
Sources: docs-website/static/llms.txt:115, metadata-ingestion/recipe_overview.md:12-18
Types of Automations
DataHub provides several built-in automations that handle common metadata management tasks. These are categorized into propagation, synchronization, and AI-driven enhancements.
Metadata Propagation
Propagation automations ensure that context added at a high level (like a Domain or Business Glossary) flows down to individual datasets and columns.
- Documentation Propagation: Automatically copies descriptions from upstream to downstream entities.
- Glossary Term Propagation: Ensures standardized business terminology is applied consistently throughout the lineage.
Sources: docs-website/sidebars.js:238-243
Cloud-Specific Automations
DataHub Cloud introduces advanced automation capabilities that leverage managed services and AI:
- AI Documentation: Auto-generates table and column descriptions using LLMs.
- Metadata Sync: Specific connectors for BigQuery, Databricks, and Snowflake that keep DataHub in sync with source platform metadata automatically.
- Approval Workflows: Structured flows for approving changes to critical business metrics or glossary definitions.
Sources: docs-website/sidebars.js:244-261, docs-website/src/learn/business-metric.md:73-77
Action Workflows
Action workflows represent a more structured approach to automation, particularly for data access and lifecycle management. These are integrated into the UI for easier management.
This sequence diagram details the lifecycle of a workflow from configuration to execution.
Sources: docs-website/sidebars.js:200-215, docs-website/generateDocsDir.ts:717-724
Technical Implementation
Automations are often configured via recipes or specific plugin implementations. The framework relies on the DataHub Metadata Change Log (MCL) as the primary event source.
Transformers in Automations
Transformers act as middleware within the ingestion and automation pipeline, allowing for the modification of metadata before it reaches the sink. A common use case is adding default tags or owners during ingestion.
# Example Transformer configuration in a recipe
transformers:
- type: simple_add_dataset_tags
config:
tag_urns:
- "urn:li:tag:Important"
Sources: metadata-ingestion/recipe_overview.md:100-117
Connector Development Standards
When building custom actions or connectors, DataHub provides "Skills" via AI-assisted frameworks to ensure compliance with architectural standards, including error handling, type safety, and reporting.
- Load Standards: Loads base classes and SDK V2 patterns.
- Review Connector: Validates architecture against production standards before deployment.
Sources: metadata-ingestion/datahub-skills.md:28-35, metadata-ingestion/adding-source.md:11-15
Summary
DataHub Actions & Automations transform DataHub from a passive catalog into an active metadata platform. By leveraging event-driven architecture, the framework automates documentation, governance, and communication workflows. This reduces the manual overhead of metadata maintenance and ensures that stakeholders are notified of critical changes in real-time, fostering a more reliable and transparent data ecosystem.
Sources: docs-website/static/llms.txt:5-10, docs-website/sidebars.js:235-237