REST & OpenAPI Endpoints
Relevant source files
The following files were used as context for generating this wiki page:
DataHub provides a set of RESTful APIs and OpenAPI endpoints to facilitate programmatic interaction with the metadata graph. These endpoints allow for the discovery, governance, and observability of data assets by human teams and AI agents alike. The primary interface for these services is located under the /openapi/v3 path prefix, offering a modern, versioned approach to entity management.
The OpenAPI implementation serves as a robust alternative to DataHub's GraphQL and Rest.li APIs, providing standardized endpoints for entity manipulation, batch operations, and metadata scrolling. It is designed to be accessible via standard HTTP clients and SDKs, supporting both synchronous and asynchronous ingestion patterns.
Sources: metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/v3/OpenAPIV3Generator.java:38-42, docs-website/static/llms.txt:53-58
API Architecture and Generation
The OpenAPI specification in DataHub is dynamically generated based on the project's EntityRegistry. The OpenAPIV3Generator class handles the construction of the OpenAPI 3.1.0 specification, translating DataHub's internal entity and aspect models into standard OpenAPI schemas.
Specification Generation Flow
The generator iterates through filtered entity and aspect specs, building request and response schemas for each. It handles complex tasks such as translating Avro schemas to JSON schemas and managing nullability for OpenAPI 3.1.0 compatibility.
The diagram above illustrates the high-level flow of the OpenAPI specification generation process.
Sources: metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/v3/OpenAPIV3Generator.java:60-150, docs-website/generateDocsDir.ts:400-420
Core Endpoint Categories
DataHub organizes its REST endpoints into specific categories based on the level of abstraction and the target entity type.
Generic Entity Operations
These endpoints provide a unified way to interact with any entity type within DataHub without requiring entity-specific knowledge in the URL path.
| Endpoint Path | Method | Description |
|---|---|---|
/openapi/v3/entity/scroll | POST | Scroll across all indexed entities and aspects. |
/openapi/v3/entity/generic | POST | Create or update (upsert) generic entities. |
/openapi/v3/entity/generic | PATCH | Apply partial updates to generic entities using JSON patch. |
/openapi/v3/entity/generic/{urn} | GET | Retrieve a specific entity by its URN. |
/openapi/v3/entity/generic/{urn} | DELETE | Remove an entity and its associated aspects. |
Entity-Specific Operations
For more structured interactions, DataHub exposes endpoints tailored to specific entity types defined in the registry (e.g., datasets, users, dashboards).
| Endpoint Path Pattern | Methods | Purpose |
|---|---|---|
/entity/{entityName} | GET, POST, PATCH | List, batch create, or batch patch specific entity types. |
/entity/{entityName}/{urn} | GET, HEAD, DELETE | Single entity CRUD and existence checks. |
/entity/{entityName}/batchGet | POST | Retrieve multiple instances of a specific entity type. |
/entity/{entityName}/{urn}/{aspectName} | GET, POST, PATCH, DELETE | Direct manipulation of specific aspects on an entity. |
Data Interaction Models
DataHub uses standardized structures for requests and responses to ensure consistency across the API.
Request Components
Requests often include parameters for system metadata, soft-delete inclusion, and pagination.
- systemMetadata: Boolean flag to include system-level metadata in the response.
- includeSoftDelete: Boolean flag to include items that have been logically deleted.
- aspects: A list of specific aspects to retrieve for the entity.
The sequence diagram shows a typical batch retrieval flow for specific entity types.
Sources: metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/v3/OpenAPIV3Generator.java:298-320, metadata-ingestion/adding-source.md:10-30
Patching and Versioning
The API supports complex metadata updates through patching and provides specialized endpoints for versioning relationships.
- Aspect Patching: Uses a custom JSON Patch format to manipulate array sets representing maps with primary keys.
- Versioning Relationship: Specific endpoints under
/entity/versioning/{versionSetUrn}/relationship/versionOf/{entityUrn}manage the unlinking and linking of entities within version sets.
Sources: metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/v3/OpenAPIV3Generator.java:995-1050, metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/v3/OpenAPIV3Generator.java:1115-1150
Implementation Details
The following code snippet demonstrates how the OpenAPIV3Generator defines the schema for an aspect reference in a response, ensuring it includes system metadata and audit stamps when required.
// Sources: [metadata-service/openapi-servlet/src/main/java/io/datahubproject/openapi/v3/OpenAPIV3Generator.java:605-630]
private static Schema buildAspectRefResponseSchema(final String aspectName) {
final Schema result = newSchema()
.type(TYPE_OBJECT)
.description(ASPECT_DESCRIPTION)
.required(List.of(PROPERTY_VALUE))
.addProperty(PROPERTY_VALUE, newSchema().$ref(PATH_DEFINITIONS + aspectName));
result.addProperty(NAME_SYSTEM_METADATA, newSchema()
.types(TYPE_OBJECT_NULLABLE)
.oneOf(List.of(newSchema().$ref(PATH_DEFINITIONS + "SystemMetadata"), newSchema().type(TYPE_NULL))));
result.addProperty(NAME_AUDIT_STAMP, newSchema()
.types(TYPE_OBJECT_NULLABLE)
.oneOf(List.of(newSchema().$ref(PATH_DEFINITIONS + "AuditStamp"), newSchema().type(TYPE_NULL))));
return result;
}
Summary
The REST and OpenAPI endpoints in DataHub provide a comprehensive, programmatically accessible layer for managing the metadata graph. By dynamically generating the specification from the EntityRegistry, DataHub ensures that the API remains in sync with the evolving metadata model, supporting everything from simple entity lookups to complex batch updates and metadata scrolling.