Search & Graph Indices
Relevant source files
The following files were used as context for generating this wiki page:
Search and Graph Indices form the backbone of DataHub's metadata discovery and relationship traversal capabilities. These indices allow for high-performance querying of entities, their attributes, and the complex web of lineages and associations that connect them. The system primarily leverages Elasticsearch (or OpenSearch) for search indexing and structured graph storage to power features like Search and Lineage.
Sources: docs-website/sidebars.js:335, metadata-service/restli-api/src/main/snapshot/com.linkedin.operations.operations.snapshot.json:2006
Architecture and Data Flow
DataHub employs an event-driven architecture where metadata changes are propagated to the Search and Graph indices. When a metadata aspect is updated, the system triggers indexing operations to ensure the Search and Graph views remain synchronized with the primary metadata store.
Indexing Flow
The indexing process transforms raw metadata aspects into searchable documents and graph edges. This involves:
- Metadata Ingestion: Metadata is ingested via MCE (Metadata Change Events) or MCP (Metadata Change Proposals).
- Document Transformation: Aspects are mapped to specific search fields (e.g.,
TEXT,KEYWORD,URN). - Graph Edge Creation: Relationships defined in aspects (like
ConsumesorOwnedBy) are materialized as edges in the graph index.
The diagram shows how ingested metadata is routed to both Search and Graph indices to support different API consumers.
Sources: metadata-ingestion/adding-source.md:63, metadata-service/restli-api/src/main/snapshot/com.linkedin.operations.operations.snapshot.json:2006
Search Indexing Configuration
Search indices are configured via annotations in the metadata model. These annotations define how fields should be indexed, their boost scores, and whether they support autocomplete.
Searchable Field Types
DataHub supports various field types for indexing:
- TEXT / WORD_GRAM: Used for full-text search and partial matches.
- KEYWORD: Used for exact matches and filtering.
- URN: Specifically optimized for unique identifiers.
- DATETIME: Supports range queries for timestamps like
createdAtorlastModifiedAt.
| Feature | Description | File Context |
|---|---|---|
| Boost Score | Assigns importance to specific fields (e.g., Entity Name has higher weight). | operations.snapshot.json:43 |
| Autocomplete | Enables suggestions during user input. | operations.snapshot.json:44 |
| Search Tier | Categorizes fields for query prioritization (Tiers 1-4). | operations.snapshot.json:47 |
Sources: metadata-service/restli-api/src/main/snapshot/com.linkedin.operations.operations.snapshot.json:41-55
Graph and Relationship Indices
The Graph index manages the relationships between entities. Relationships are defined within aspects using the @Relationship annotation, which specifies the edge name and the destination entity types.
Key Graph Relationships
Common relationships indexed include:
- Consumes: Lineage relationship between DataJobs and Datasets.
- OwnedBy: Association between an entity and its owners (CorpUsers/Groups).
- IsPartOf: Hierarchical relationship for Containers and Glossaries.
The ER diagram illustrates the core entity relationships stored in the graph index.
Sources: metadata-service/restli-api/src/main/snapshot/com.linkedin.operations.operations.snapshot.json:115, metadata-service/restli-api/src/main/snapshot/com.linkedin.operations.operations.snapshot.json:549
Maintenance and Operations API
DataHub provides specialized endpoints for managing the health and state of Search and Graph indices. These are accessible via the Operations resource.
Operations Actions
- restoreIndices: Rebuilds search and graph indices from the primary metadata store. Parameters include
urn,urnLike, andaspectfor targeted restoration. - getIndexSizes: Returns the storage size for various indices, categorized by entity and aspect.
- getEsTaskStatus: Monitors the progress of long-running Elasticsearch tasks, such as reindexing.
- truncateTimeseriesAspect: Manages the lifecycle of time-series data by removing older entries based on
endTimeMillis.
This sequence demonstrates the operational flow for restoring indices and monitoring the task status.
Sources: metadata-service/restli-api/src/main/snapshot/com.linkedin.operations.operations.snapshot.json:2006-2090
Vector Indices (Experimental/Connector-specific)
With the rise of Vector Databases like Pinecone, DataHub's indexing capabilities are extending to handle vector metadata. These indices track dimensions, similarity metrics, and namespaces within vector databases to provide discovery for ML assets.
Vector Metadata Fields
| Field | Description |
|---|---|
dimension | The size of the vector embedding. |
metric | Similarity metric used (e.g., Cosine, Euclidean). |
namespace | Logical partitions within a vector index. |
Conclusion
Search and Graph Indices are critical components that enable DataHub's high-level features. Through structured indexing configurations and a robust operations API, the system ensures that metadata is not only stored but is also highly discoverable and traversable. Regular maintenance using the Operations API is recommended to keep indices healthy and performant.