Search & Discovery
Relevant source files
The following files were used as context for generating this wiki page:
DataHub's Search & Discovery system provides a unified interface for exploring a vast metadata graph that includes datasets, dashboards, ML models, and data pipelines. It is designed to help users find any data asset across their entire tech stack through high-performance full-text search and intuitive browsing experiences.
The system bridges the gap between technical metadata and business context, allowing users to discover assets based on technical properties, ownership, tags, and glossary terms. This functionality is delivered via a React-based frontend that communicates with a GraphQL API layer, which in turn interacts with the Metadata Service (GMS).
Sources: docs-website/static/llms.txt, datahub-web-react/README.md
Architecture and Data Flow
The search architecture follows a multi-tier approach where user queries from the React UI are processed by a GraphQL engine before reaching the search indices.
Search Flow Overview
The following diagram illustrates the path of a search request from the user interface to the metadata graph.
The search request is handled by the SearchAcrossEntitiesResolver in the GraphQL layer, which enables unified search across all DataHub entities. Results returned as URNs are mapped to GraphQL entities via the UrnToEntityMapper.
Sources: datahub-graphql-core/README.md:94-105
Core Search Capabilities
DataHub supports several discovery modes to accommodate different user needs, from open-ended exploration to specific asset lookups.
| Feature | Description |
|---|---|
| Full-Text Search | Search for datasets, dashboards, and pipelines using keywords against metadata. |
| Advanced Filters | Specific search queries using Type, Owner, Tag, and more. |
| Browse-by-Path | Navigate metadata in a filesystem-like hierarchy. |
| Ask DataHub | Natural-language search across metadata (Cloud feature). |
| Explore All | A starting point for users unsure of specific keywords. |
Sources: docs-website/static/llms.txt, datahub-web-react/src/i18n/locales/en/onboarding.json, datahub-web-react/README.md
Search and Browse in React
The React application organizes search functionality into dedicated modules. The search module contains components for the full-text search experience, while the browse module handles the filesystem-like hierarchy navigation. Previews of entities are rendered using shared components that adapt based on the context (e.g., SEARCH vs BROWSE vs PREVIEW).
Sources: datahub-web-react/README.md:129-142
Enabling Search for New Entities
To make a new entity discoverable via search, it must be integrated into the GraphQL core and the unified search resolver.
Implementation Steps
- GraphQL Definition: The entity must extend the
Entityinterface inentity.graphqland include a top-level Query for primary key lookups using an "urn" parameter. - Resolver Registration: The new Entity type must be added to the list in
SearchAcrossEntitiesResolver.java. - URN Mapping: A statement must be added to
UrnToEntityMapper.javato map the result URN to a placeholder GraphQL entity.
Sources: datahub-graphql-core/README.md:88-105
Discovery and Exploration Tools
Beyond simple search, DataHub provides tools to browse the data estate by organizational context.
Domains and Platforms
- Domains: Collections of related data assets associated with specific organizational units (e.g., Marketing). Users can "Explore by Domain" to find relevant assets.
- Platforms: Represent third-party data systems (e.g., Snowflake, Airflow, Looker). Users can "Explore by Platform" to see all assets within a specific tool.
Sources: datahub-web-react/src/i18n/locales/en/onboarding.json
Business Glossary
The Business Glossary serves as a discovery tool by providing standardized labels (Terms and Term Groups). Users can find mission-critical data by searching for these business definitions rather than technical names.
Sources: datahub-web-react/src/i18n/locales/en/onboarding.json
Configuration and Customization
The search experience is configurable through various environment variables and UI settings.
| Variable | Purpose |
|---|---|
REACT_APP_LOGO_URL | Customizes the logo in the search bar header. |
search.advanced | Enables/Disables advanced filter capabilities in the UI. |
search.browse | Controls visibility of inline platform navigation. |
Organizations can also customize the "Most Popular" exploration section, which surfaces assets viewed most frequently.
Sources: datahub-web-react/README.md:92-95, datahub-web-react/src/i18n/locales/en/onboarding.json
Summary
The Search & Discovery system in DataHub is a comprehensive framework that integrates high-level business abstractions (Domains, Glossary Terms) with low-level technical metadata. By utilizing a GraphQL-driven architecture and a unified search resolver, it provides a consistent experience across all entity types, ensuring that users can confidently manage breaking changes and understand data origins through end-to-end discovery.