Kubernetes Deployment
Relevant source files
The following files were used as context for generating this wiki page:
- datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java
- metadata-service/openapi-servlet/src/test/java/io/datahubproject/openapi/operations/k8/KubernetesControllerTest.java
- docs-website/sidebars.js
- docs-website/static/llms.txt
- docs-website/generateDocsDir.ts
- metadata-ingestion/CLAUDE.md
Kubernetes is a primary deployment target for DataHub, providing a unified platform for discovery, governance, and observability across the data stack. The deployment architecture leverages standard Kubernetes primitives—such as Deployments, Pods, ConfigMaps, and Jobs—to manage the lifecycle of DataHub components like the Metadata Service (GMS), the Frontend UI, and various consumers.
The system includes dedicated utilities for managing in-cluster operations, such as scaling workloads, performing system updates, and interacting with the Kubernetes API to monitor rollout status.
Sources: docs-website/static/llms.txt:1-10, datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java:31-35
Core Architecture and API Access
DataHub interacts with the Kubernetes API using a specialized accessor and controller to perform administrative tasks within the cluster.
Kubernetes API Accessor
The KubernetesApiAccessor utilizes the Fabric8 client to interface with the cluster. It is primarily used during scale-down steps and cleanup operations during system upgrades. It can detect if it is running inside a pod by checking for the KUBERNETES_SERVICE_HOST environment variable and can automatically resolve the namespace from the service account file.
Key capabilities include:
- Rollout Monitoring: Polling deployments to ensure that new specifications are fully reconciled and that desired replica counts match available and ready counts.
- Environment Management: Editing container environment variables in-place to minimize API server diffs and avoid destroying secret/configmap bindings.
- State Persistence: Using ConfigMaps to store scale-down retry and restore values during upgrades.
Sources: datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java:63-100, datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java:232-260, datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java:444-468
Kubernetes Operations Controller
The system exposes an OpenAPI-compliant controller for Kubernetes operations, enabling external management of cluster resources. This controller supports operations like listing deployments, scaling replicas, retrieving pod logs, and triggering CronJobs.
API Endpoints Summary
| Method | Endpoint | Description |
|---|---|---|
| GET | /status | Returns the availability and current namespace of the K8s interface. |
| GET | /deployments | Lists all deployments in the namespace with pagination support. |
| PATCH | /deployments/{name}/scale | Scales a deployment's replicas or updates resource limits. |
| GET | /pods/{name}/logs | Streams logs from a specific pod and container. |
| POST | /cronjobs/{name}/trigger | Manually triggers a new Job from a CronJob template. |
Sources: metadata-service/openapi-servlet/src/test/java/io/datahubproject/openapi/operations/k8/KubernetesControllerTest.java:95-101, metadata-service/openapi-servlet/src/test/java/io/datahubproject/openapi/operations/k8/KubernetesControllerTest.java:400-410, metadata-service/openapi-servlet/src/test/java/io/datahubproject/openapi/operations/k8/KubernetesControllerTest.java:480-490
Workload Management and Scaling
Deployment Rollout Logic
DataHub implements strict logic to verify when a deployment rollout is complete. A rollout is considered finished only when the Deployment controller has applied the latest specification and no stale pods remain.
The logic ensures observedGeneration >= metadata.generation and that all replica counters (current, updated, available, and ready) match the desired count.
KEDA Autoscaling Integration
DataHub supports integration with KEDA (Kubernetes Event-driven Autoscaling). During operations that require manual scaling or pausing of consumers, the system can interact with KEDA ScaledObject resources.
If a deployment is managed by KEDA, DataHub can "pause" the autoscaler by applying the autoscaling.keda.sh/paused-replicas annotation. Scaling operations first attempt to find a related ScaledObject before falling back to direct deployment scaling.
Sources: datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java:356-373, metadata-service/openapi-servlet/src/test/java/io/datahubproject/openapi/operations/k8/KubernetesControllerTest.java:1075-1090
Configuration and State Management
Kubernetes ConfigMaps are used to manage the state of system updates. This prevents data loss or inconsistent states if an upgrade process is interrupted and needs to be retried.
Scale-Down State
The system resolves the ConfigMap name using the HELM_RELEASE_NAME environment variable, defaulting to datahub-system-update-scale-down-state. This ConfigMap persists a JSON-serialized ScaleDownState object containing:
- Retry Counts: Tracking attempt numbers.
- Restore Values: Storing the original replica counts before scaling down to zero for maintenance.
Sources: datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java:128-150, datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/kubernetes/KubernetesApiAccessor.java:433-442
Resource Configuration Table
| Configuration Property | Type | Default | Description |
|---|---|---|---|
rolloutMaxWaitSeconds | Integer | Required | Maximum time to wait for a deployment rollout. |
rolloutPollSeconds | Integer | Required | Interval between polling for rollout status. |
kedaGroup | String | Required | API group for KEDA (e.g., keda.sh). |
stateConfigMapName | String | Optional | Override for the state persistence ConfigMap. |
Execution and Cleanup
The KubernetesApiAccessor provides utility methods to clean up resources after system updates, particularly active Batch Jobs.
The cleanup process ignores jobs containing the string system-update to avoid deleting the active upgrade executor.
Conclusion
Kubernetes deployment in DataHub provides a robust framework for managing complex metadata microservices. By utilizing a dedicated API accessor and controller, the project ensures that system upgrades, workload scaling, and resource monitoring are handled with awareness of in-cluster state and third-party autoscalers like KEDA. This integration allows DataHub to maintain high availability and predictable rollout behavior across diverse Kubernetes environments.