Docker Deployment

Relevant source files

The following files were used as context for generating this wiki page:

• docker/README.md
• docker/docker-compose.yml
• docs-website/sidebars.js
• metadata-ingestion/adding-source.md
• datahub-web-react/README.md
• metadata-ingestion-modules/airflow-plugin/DOCKER_TEST_GUIDE.md

Docker Deployment is the primary mechanism for local development, quick testing, and containerized production hosting of the DataHub platform. It utilizes a microservices architecture where core components—such as the Generalized Metadata Service (GMS), frontend UI, and consumers—are distributed as individual Docker images. These images are continuously deployed to Docker Hub with every repository commit.

The deployment infrastructure leverages Docker Compose profiles to manage complex dependencies like Elasticsearch, MySQL/PostgreSQL, and Kafka. While quickstart tags are provided for rapid local setup, production environments typically pin specific release or immutable commit tags for stability.

Sources: docker/README.md:14-19, docker/README.md:27-31

Core Component Architecture

DataHub's Docker architecture consists of several specialized service containers and infrastructure dependencies.

Service Containers

The following table describes the primary DataHub service images:

Service Image	Purpose
datahub-gms	The Generalized Metadata Service; core backend API.
datahub-frontend-react	The React-based user interface.
datahub-mae-consumer	Metadata Audit Event consumer for search/graph updates.
datahub-mce-consumer	Metadata Change Event consumer for ingestion processing.
datahub-ingestion	CLI-based container for running metadata ingestion jobs.
datahub-upgrade	Handles system updates, SQL setup, and index management.
datahub-actions	Event-driven automation framework for metadata changes.

Sources: docker/README.md:33-40, datahub-web-react/README.md:10-14

Infrastructure Requirements

Running the full DataHub stack via Docker requires specific resource allocations and backing services:

• Hardware: Minimum 2 CPUs, 8GB RAM, and 2GB Swap.
• Backing Services: Elasticsearch (or OpenSearch), MySQL (or PostgreSQL), and optionally Neo4j.
• Setup: The datahub-upgrade job (with -u SystemUpdate) performs SQL and search index initialization once backing services are healthy.

Sources: docker/README.md:8-10, docker/README.md:105-110

Component Relationship Diagram

The following diagram illustrates the interaction between DataHub containers and external storage systems.

Image Variants and Tags

DataHub provides specialized image variants for datahub-ingestion and datahub-actions to balance compatibility and image size.

Variant Types

• Full: Contains all connectors and maximum compatibility, including Oracle and MSSQL drivers.
• Slim: Includes common connectors; recommended for standard cloud data stacks.
• Locked: For air-gapped environments where runtime pip install is prohibited (PyPI disabled).

Sources: docker/README.md:46-52

Tagging Strategy

Deployment configurations must use appropriate tags based on the environment:

• Local/Quickstart: Uses the quickstart tag or coordinated versions (e.g., v0.8.40).
• Production/Kubernetes: Pin a release tag (v*) or an immutable commit tag (sha-<short_sha>).
• Deprecation Note: The latest and debug tags are unsupported.

Sources: docker/README.md:27-31

Deployment Workflows

Quickstart Workflow

The datahub docker quickstart command is the entry point for open-source users. It downloads a flattened compose file from the repository.

Nuke Tasks (Cleanup)

DataHub provides targeted cleanup tasks via Gradle to reset specific environments:

• quickstartNuke: Removes containers and volumes for the default datahub namespace.
• quickstartCypressNuke: Cleans up isolated integration test environments (dh-cypress).
• quickstartPgNuke: Targets quickstart-postgres configurations.

Sources: docker/README.md:133-145

Development Environment Setup

Developers can run the UI or Ingestion framework against a Docker-hosted backend.

UI Development

To develop the frontend without rebuilding Docker images:

1. Deploy the backend services via Docker (typically at http://localhost:9002).
2. Run yarn install && yarn run start in the datahub-web-react directory.
3. The local server proxies GraphQL requests to the datahub-frontend container.

Sources: datahub-web-react/README.md:52-60

Integration Testing in Docker

For Airflow plugin testing, DataHub uses a dedicated Dockerfile.test and docker-compose.test.yml. This allows running integration tests in an environment that mimics production Airflow without local Python dependency conflicts.

# Example Airflow plugin test execution
docker-compose -f docker-compose.test.yml run --rm airflow-plugin-test py311-airflow31

Sources: metadata-ingestion-modules/airflow-plugin/DOCKER_TEST_GUIDE.md:20-25

Summary

Docker Deployment provides a portable, standardized method for operating the DataHub metadata platform. By using a combination of specialized service images, variant types (Full, Slim, Locked), and Docker Compose profiles, the project supports a wide range of use cases from local development to air-gapped production hosting. Key management tasks, including service initialization and environment cleanup, are automated through the datahub-upgrade container and Gradle-based nuke tasks.