Data Quality & Assertions
Relevant source files
The following files were used as context for generating this wiki page:
- datahub-web-react/src/app/entityV2/dataset/profile/tabs/Validations/AcrylValidationsTab.tsx
- metadata-service/services/src/main/java/com/linkedin/metadata/service/AssertionService.java
- docs-website/sidebars.js
- docs-website/src/learn/data-freshness.md
- metadata-ingestion/src/datahub/ingestion/source/odcs/odcs_schema/odcs-v3.1.0.json
- docs-website/static/llms.txt
- metadata-ingestion/adding-source.md
Data Quality & Assertions in DataHub comprise a unified framework for monitoring and maintaining the reliability of data assets across the data estate. This system allows data owners and consumers to define, execute, and visualize validation checks—known as Assertions—to ensure data meets specific standards for freshness, volume, schema integrity, and column-level statistics. By integrating these checks into the metadata graph, DataHub facilitates proactive discovery of quality issues before they impact downstream consumers or AI models. Sources: docs-website/static/llms.txt:38-38, docs-website/src/learn/data-freshness.md:7-12
The system supports automated anomaly detection, manual assertion definitions, and integration with the Open Data Contract Standard (ODCS) to establish formal agreements between data producers and consumers. Assertions provide visibility into data health through the DataHub UI and support notifications via SMTP or Slack when quality standards are violated. Sources: docs-website/sidebars.js:216-224, docs-website/static/llms.txt:64-64
Assertion Categories and Types
DataHub categorizes assertions based on the dimension of data quality they address. These assertions can be configured via the DataHub UI or programmatically using the Python and Java SDKs.
Standard Assertion Types
| Assertion Type | Description | Key Capabilities |
|---|---|---|
| Column Assertions | Validates field-level data (e.g., value ranges, null counts). | Ensures specific fields adhere to business logic. |
| Custom SQL Assertions | Executes arbitrary SQL queries to validate complex logic. | Highly flexible for system-specific checks. |
| Freshness Assertions | Monitors when data was last updated against an SLA. | Detects stale data using Audit Logs or event times. |
| Schema Assertions | Monitors for structural changes in datasets. | Alerts on field additions, deletions, or type changes. |
| Volume Assertions | Monitors row counts and storage size changes. | Detects unexpected drops or spikes in data volume. |
| Anomaly Detection | Automated statistical monitoring (Smart Assertions). | Detects outliers based on historical patterns. |
Sources: docs-website/sidebars.js:216-258, docs-website/src/learn/data-freshness.md:95-103
Open Data Contract Standard (ODCS) Integration
DataHub leverages the ODCS schema to define data quality checks as part of a formal data contract. This schema allows developers to specify human-readable descriptions, technical metrics, and comparison operators.
Data Quality Dimensions in ODCS
The ODCS specification identifies seven primary dimensions for data quality checks:
- Accuracy: Correctness of the data values.
- Completeness: Presence of required data (e.g., null checks).
- Conformity: Adherence to specific formats.
- Consistency: Uniformity across different systems or tables.
- Coverage: Extent of data availability.
- Timeliness: Freshness and latency.
- Uniqueness: Lack of duplicate records.
Sources: metadata-ingestion/src/datahub/ingestion/source/odcs/odcs_schema/odcs-v3.1.0.json:886-886
ODCS Quality Rule Structure
Quality rules in ODCS are defined using a structured DataQuality object which includes the check type and implementation details.
The diagram above shows the hierarchy of data quality checks supported by the ODCS schema integration in DataHub. Sources: metadata-ingestion/src/datahub/ingestion/source/odcs/odcs_schema/odcs-v3.1.0.json:871-932
Data Freshness & Monitoring
Data freshness is a critical subset of assertions that measures the timeliness of data. Freshness is calculated by comparing the time an event occurred versus when it was reflected in the target dataset. Sources: docs-website/src/learn/data-freshness.md:15-18
Freshness Assertions
Freshness assertions allow users to set Service Level Agreements (SLAs). DataHub monitors these by:
- Tracking Lineage: Identifying the "supply chain" of data to pinpoint where delays occur. Sources: docs-website/src/learn/data-freshness.md:73-77
- Audit Log Monitoring: Inspecting system logs (e.g., Snowflake Query History) to determine the last update time. Sources: docs-website/src/learn/data-freshness.md:118-121
- Anomaly Detection: Using "Smart Assertions" to automatically detect changes in the update cadence based on historical data. Sources: docs-website/src/learn/data-freshness.md:118-121
Freshness Monitoring Flow
This sequence illustrates how metadata extraction flows through the system to update freshness status and trigger alerts. Sources: docs-website/src/learn/data-freshness.md:114-121, metadata-ingestion/adding-source.md:118-125
Operational Management
Incident Management
When an assertion fails, DataHub generates an Incident. This allows teams to:
- Triage and prioritize quality issues.
- Acknowledge and assign owners to failures.
- Communicate status to downstream stakeholders using lineage-based impact analysis.
Sources: docs-website/src/learn/data-freshness.md:130-134, docs-website/static/llms.txt:67-67
Assertion Configuration (Python Config Example)
Connectors must declare support for quality-related capabilities (e.g., DATA_PROFILING) to enable assertion execution.
# Sources: [metadata-ingestion/adding-source.md:90-103]
@platform_name("File")
@support_status(SupportStatus.CERTIFIED)
@config_class(FileSourceConfig)
@capability(SourceCapability.DATA_PROFILING, "Optionally enabled via configuration")
class FileSource(Source):
"""
The File Source produces metadata that assertions can validate.
"""
Summary
Data Quality & Assertions serve as the observability layer of DataHub, bridging the gap between static metadata and live data health. By utilizing standard metrics (ODCS), automated anomaly detection, and end-to-end lineage, the system provides a robust framework for managing data reliability and enforcing organizational SLAs. Sources: docs-website/src/learn/data-freshness.md:144-148