Search Integration
Relevant source files
The following files were used as context for generating this wiki page:
The search functionality for the Datadog documentation website is a sophisticated system designed to provide fast, relevant, and section-specific results. It leverages the Hugo static site generator for index creation, a dedicated search backend service (Typesense), and a powerful client-side library (InstantSearch.js) to render the user interface. This integration ensures that users can efficiently navigate the vast amount of documentation, including product-specific query syntaxes for Logs, Metrics, and Traces.
The architecture is decoupled, with Hugo responsible for preparing the data, the search service responsible for indexing and querying, and the frontend JavaScript responsible for the user experience. This allows for a robust and scalable search solution integrated directly into the static documentation site.
Architecture Overview
The search system follows a multi-stage process, from content creation to displaying results to the user. The core components are the Hugo build process, the search index file, the Typesense search backend, and the InstantSearch.js frontend.
The following diagram illustrates the high-level data flow of the search integration.
Index Generation
The search index is generated as a JSON file (search.json) during the Hugo build process. This process is orchestrated by the template file layouts/_default/list.search.json and is triggered only in specific CI/CD environments, such as live or typesense_sync_preview:manual.
Sources: layouts/_default/list.search.json:1-2
Page Processing
The build script iterates through all site pages and filters them to include only relevant documentation content. It excludes pages like partner pages, private content, drafts, and external redirects. For each valid page, it extracts key metadata.
Sources: layouts/_default/list.search.json:6-9
Content Chunking
To provide more granular search results that link directly to a specific section of a document, pages are broken down into smaller records. The layouts/partials/algolia/page-sections.json partial is responsible for splitting the page content by <h2> headings. Each resulting section becomes a separate entry in the search index. If a page contains no <h2> headings, the entire page is indexed as a single record.
This process creates a structured record for each section, which is then added to the final search.json file.
Sources: layouts/_default/list.search.json:30-32, layouts/partials/algolia/page-sections.json
Index Record Schema
Each record in the generated search.json file corresponds to a page section and has a well-defined structure.
The following diagram shows the relationship between a Page and its generated Section Records.
The key fields for each index record are detailed in the table below.
| Field | Type | Description | Source |
|---|---|---|---|
objectID | string | A unique identifier for the record, generated from an MD5 hash of the page path and the section anchor. | layouts/partials/algolia/page-sections.json:11 |
title | string | The main title of the source page. | layouts/partials/algolia/page-sections.json:12 |
section_header | string | The text of the <h2> heading for this section. Empty for full-page records. | layouts/partials/algolia/page-sections.json:14 |
content | string | The plain text content of the section, truncated to 10,000 characters. | layouts/_default/list.search.json:13 |
relpermalink | string | The relative URL of the page, including the section anchor. | layouts/partials/algolia/page-sections.json:18 |
full_url | string | The absolute URL of the page. | layouts/partials/algolia/page-sections.json:20 |
language | string | The language code of the page (e.g., en). | layouts/partials/algolia/page-sections.json:21 |
category | string | The primary search category, typically "Documentation". | layouts/_default/list.search.json:16 |
subcategory | string | The subcategory, derived from the page's section title. | layouts/_default/list.search.json:22 |
rank | integer | A numeric value used for sorting results. Defaults to 70. | layouts/_default/list.search.json:15 |
order | integer | The order of the section within the page, starting from 0. | layouts/partials/algolia/page-sections.json:25 |
Search Backend: Typesense
The project uses Typesense as its search backend. The configuration for the Typesense collection is defined in typesense.config.json. This file specifies the schema that the search index must adhere to.
Collection Schema
The schema defines the fields, their data types, and which ones are indexed. The collection name is dynamically generated based on the site language, for example, docr-pages-en.
Sources: typesense.config.json:3
The following table outlines the schema configuration.
| Field Name | Type | Indexed | Optional | Facet |
|---|---|---|---|---|
title | string | Yes | No | No |
section_header | string | Yes | No | No |
content | string | Yes | No | No |
category | string | Yes | No | Yes |
subcategory | string | Yes | No | Yes |
tags | string[] | Yes | Yes | Yes |
language | string | Yes | No | Yes |
full_url | string | No | No | No |
relpermalink | string | No | No | No |
order | int32 | Yes | No | No |
rank | int32 | Yes | No | No |
The default_sorting_field is set to rank, meaning results are primarily sorted by this value in descending order.
Sources: typesense.config.json:4-47
Frontend Implementation
The search user interface is built using HTML from layouts/partials/search.html and powered by JavaScript in assets/scripts/components/instantsearch.js. It utilizes the InstantSearch.js library, which, despite being from Algolia, is connected to the Typesense backend via the typesense-instantsearch-adapter.
Sources: assets/scripts/components/instantsearch.js:1, 10
UI Components
The search interface is a modal dialog containing several key widgets.
The main widgets are:
searchBox: The main text input for user queries, placed in the#searchboxelement.hits: Displays the search results. The template for each hit is custom-defined to show the page title, section header, and a snippet of the content with matching terms highlighted.refinementList: A filter for search results, used forcategoryandsubcategoryfacets.configure: Used to set search parameters likehitsPerPage.poweredBy: Displays a "Powered by" logo.
Sources: assets/scripts/components/instantsearch.js:38-142, layouts/partials/search.html:12-28
Result Rendering
The hits widget uses a custom template to format each search result. It constructs a breadcrumb-style navigation path using the category, subcategory, and title. The section_header is displayed prominently if it exists, and a content snippet shows the context of the search match.
// assets/scripts/components/instantsearch.js:63-107
// ...
templates: {
item(hit, { html, components }) {
// ...
const breadcrumbs = [
hit.category,
hit.subcategory,
hit.title,
]
// ...
return html`
<a href="${hit.relpermalink}" class="search-result-link">
<div class="search-result">
<div class="search-result-page-icon">...</div>
<div class="search-result-content-wrapper">
<div class="search-result-breadcrumbs">...</div>
<div class="search-result-title">
${
hit.section_header &&
html`<span class="search-result-title-primary">${components.Highlight({
hit,
attribute: 'section_header',
})}</span>`
}
</div>
<div class="search-result-snippet">
${components.Snippet({ hit, attribute: 'content' })}
</div>
</div>
</div>
</a>
`;
},
// ...
Sources: assets/scripts/components/instantsearch.js:63-107
Conclusion
The search integration in the Datadog documentation repository is a well-architected feature that combines static site generation with a dynamic, real-time search experience. By pre-processing and chunking content during the build, it populates a powerful Typesense search index. The frontend then uses Algolia's InstantSearch.js library to provide users with a fast, interactive, and highly relevant search UI, complete with filtering and deep linking to specific document sections. This system is critical for ensuring the extensive documentation remains accessible and user-friendly.