Smoke Tests & Quality Assurance
Relevant source files
The following files were used as context for generating this wiki page:
Smoke Tests and Quality Assurance (QA) in DataHub encompass a multi-layered testing strategy designed to validate the stability, functionality, and performance of the metadata platform. The scope includes end-to-end (E2E) UI testing, metadata ingestion validation, library example verification, and core system API health checks. These tests ensure that new changes do not break existing features like search, browse, authentication, and ingestion pipelines.
Sources: smoke-test/README.md:1-5, e2e-test/ui/playwright/README.md:1-3
Testing Architecture
The QA strategy is divided into localized unit tests, integration tests, and full end-to-end smoke tests. This hierarchy allows for fast development cycles while maintaining high-confidence releases through comprehensive CI pipelines.
Test Hierarchy Flow
The following diagram illustrates the relationship between different test categories and the DataHub components they validate.
Python Smoke Tests
The smoke-test directory contains the primary end-to-end validation suite. These tests are written in Python using pytest and target a running DataHub instance.
Test Categories
| Category | Target | Description |
|---|---|---|
| System Info | /openapi/v1/system-info | Validates Spring components and system properties. |
| Core E2E | Full Pipeline | Tests requiring data ingestion, Kafka, and Schema Registry. |
| Library Examples | metadata-ingestion/examples | Validates that standalone library examples work as documented. |
Sources: smoke-test/README.md:65-90, smoke-test/tests/library_examples/README.md:5-15
Execution Commands
For rapid development, specific tests or files can be targeted:
# Run all smoke tests
pytest -vv
# Run system info tests (fastest)
pytest test_system_info.py -vv
# Run specific E2E method
pytest test_e2e.py::test_gms_usage_fetch -v
Sources: smoke-test/README.md:43-52
Playwright E2E UI Testing
The modern UI testing suite utilizes Playwright (Chromium only) to validate user-facing features. This suite replaces older Cypress-based tests for critical mutations and UI workflows.
Fixture Architecture
Playwright tests use a modular fixture system to manage authentication and state:
- base-test: Auto-injects an authenticated session (usually
admin). - login-test: Provides a clean, unauthenticated context for testing login UI.
- seedingFixture: Manages per-worker test data injection, ensuring data is seeded once per feature run.
Critical UI Coverage
Key areas covered by Playwright specs include:
- Mutations: Editing documentation, managing links, and verifying sidebar visibility.
- Documents: External document profiles, last synced properties, and Notion/Confluence extraction banners.
- Ingestion V3: Creating, editing, executing, and deleting ingestion sources via the UI.
Sources: e2e-test/ui/playwright/tests/mutations-v2/v2-edit-documentation.spec.ts:10-20, e2e-test/ui/playwright/tests/documents/external-document-profile.spec.ts:5-15, e2e-test/ui/playwright/tests/ingestion-v3/sources.spec.ts:25-50
Metadata Ingestion Quality Assurance
The metadata-ingestion module maintains its own QA standards focusing on code style, type safety, and integration reliability.
Testing Conventions
- Unit Tests: Located in
tests/unit/, focusing on logic without external dependencies. - Integration Tests: Located in
tests/integration/, often requiring Docker to simulate source systems. - Golden Files: Integration tests use "golden" data files to compare ingestion outputs against expected snapshots.
Sources: metadata-ingestion/CLAUDE.md:65-75, metadata-ingestion/developing.md:313-320
Validation Tools
| Tool | Purpose | Command |
|---|---|---|
| Ruff | Formatting and Linting | ./gradlew :metadata-ingestion:lint |
| Mypy | Static Type Checking | mypy src/ tests/ |
| CheckLockFile | Dependency consistency | ./gradlew :metadata-ingestion:checkLockFile |
Sources: metadata-ingestion/CLAUDE.md:15-25, metadata-ingestion/developing.md:295-305
Library Example Integration
DataHub ensures that documentation remains accurate by running library examples as part of the integration suite. These tests validate that the example scripts in metadata-ingestion/examples/library/ are functional for end users.
Requirements for Examples
For an example to be testable, it must:
- Support Environment Variables: Read
DATAHUB_GMS_URLandDATAHUB_GMS_TOKENfrom the environment. - Exit Codes: Exit with
0on success and1on failure. - Self-Contained: Ideally create its own dependencies (e.g., creating a notebook before reading it).
Sources: smoke-test/tests/library_examples/README.md:40-65
Summary
The Smoke Tests & QA framework provides a robust safety net for the DataHub ecosystem. By combining fast Python API tests, modular Playwright UI tests, and rigorous metadata ingestion checks, the project ensures that core capabilities like data discovery and governance remain reliable across releases. Developers are encouraged to use local smoke tests for rapid feedback before submitting code to the CI pipeline.
Sources: smoke-test/README.md:120-125, e2e-test/ui/playwright/README.md:215-220