The Build Process
Relevant source files
The following files were used as context for generating this wiki page:
The build process for the Datadog documentation site is a multi-faceted system responsible for transforming Markdown content, processing assets, and auto-generating extensive API reference documentation into a static website. The system is built around the [Hugo static site generator][1], orchestrated by a combination of Node.js scripts, Python tools, and make commands. This process ensures that content from various sources, including OpenAPI specifications and external repositories, is consistently and accurately published to the documentation site.
The core of the build system handles standard static site generation, while custom scripts manage the complex task of generating API documentation from specification files. This automated approach allows for consistent and maintainable API reference pages across multiple versions. The entire process is integrated into a CI/CD pipeline that triggers automatic deployments upon merging changes to the master branch.
Sources: README.md:2, CLAUDE.md:5-6, 17-22, content/en/administrators_guide/build.md:5
Build Orchestration
The build process is orchestrated through a series of scripts defined in package.json and high-level commands available via a Makefile. These commands handle dependency installation, asset processing, static site generation with Hugo, and running the local development server.
Sources: CLAUDE.md:9-22, package.json
Key Build Commands
The primary commands for building and running the documentation site locally are managed through Yarn and Make.
| Command | Description | Source |
|---|---|---|
yarn start / make start | Executes a full build, including fetching external dependencies, and runs the development server. | CLAUDE.md:10 |
yarn build / make build | Creates a production-ready build of the static site. | CLAUDE.md:16 |
yarn run build:hugo | Runs only the Hugo build step to generate the static site from content files. | CLAUDE.md:17 |
make dependencies | Installs all project dependencies, including Node.js packages, Python packages, and external repositories. | CLAUDE.md:25 |
The following diagram illustrates the high-level flow for local development and production builds.
This diagram shows the main command flows for starting the development server and creating a production build.
Sources: CLAUDE.md:9-18, package.json
API Reference Page Generation
A critical part of the build process is the automated generation of API reference documentation from OpenAPI specification files. This is handled by the assets/scripts/build-reference-pages.js script, which processes full_spec.yaml files for different API versions (v1 and v2).
The script performs several key tasks:
- Parses and dereferences OpenAPI YAML specifications.
- Creates Markdown page structures for each API version and tag.
- Generates
examples.jsonfiles containing request/response examples. - Generates HTML tables for request/response schemas.
- Updates the API navigation menu (
menus/api.en.yaml) with entries for each API endpoint.
Sources: assets/scripts/build-reference-pages.js:10-18, 1000-1006
Generation Flow
The processSpecs function in build-reference-pages.js initiates the generation process. It reads each specified OpenAPI spec file and orchestrates a series of steps to produce the final documentation assets.
The following sequence diagram illustrates this flow:
This diagram shows the sequence of operations performed by the build-reference-pages.js script to generate API documentation from a spec file.
Sources: assets/scripts/build-reference-pages.js:1000-1020
Core Functions
The generation process is modular, with specific functions handling different parts of the task.
| Function | Description | File |
|---|---|---|
processSpecs(specs) | The main entry point. Iterates over an array of spec file paths and orchestrates the entire generation process for each. | assets/scripts/build-reference-pages.js:1000-1002 |
createPages(apiYaml, deref, apiVersion) | Creates the directory structure and _index.md files for each API tag within a specific version. | assets/scripts/build-reference-pages.js:123-125 |
createResources(apiYaml, deref, apiVersion) | Generates examples.json containing request/response examples and schema tables for each endpoint. | assets/scripts/build-reference-pages.js:154-156 |
updateMenu(specData, specs, languages) | Updates the menus/api.en.yaml file to add navigation links for all API tags and operations. | assets/scripts/build-reference-pages.js:10-13 |
schemaTable(tableType, data) | Generates an HTML table from a JSON schema object to display request or response parameters. | assets/scripts/build-reference-pages.js:885-887 |
filterExampleJson(actionType, data) | Recursively filters a schema to build a JSON example for requests, responses, or cURL commands. | assets/scripts/build-reference-pages.js:605-607 |
Generated Artifacts
The API generation script produces several types of files that are used by Hugo to build the final documentation pages.
| Artifact | Location | Purpose |
|---|---|---|
_index.md | content/en/api/<version>/<tag-slug>/ | The main Markdown file for an API section (e.g., "Dashboards"). |
examples.json | content/en/api/<version>/<tag-slug>/ | Contains JSON objects for request bodies, response bodies, and cURL examples for each endpoint in the section. |
menus/api.en.yaml | config/_default/ | The Hugo menu configuration file, updated with generated links to API sections and endpoints. |
full_spec_<version>.json | static/resources/json/ | A dereferenced, static copy of the OpenAPI spec for tools like Postman. |
Sources: assets/scripts/build-reference-pages.js:136, 191, 89, 1014
CI/CD and Deployment
The repository is configured for automatic deployment. When a pull request is merged into the master branch, a CI/CD pipeline is triggered. This process typically takes around 35 minutes to complete and results in the updated content being published to the live documentation site.
For pull requests, a branch preview is automatically generated if the branch is named correctly (<name>/<description>). This allows contributors and reviewers to preview their changes on a live, temporary site before merging.
This diagram outlines the deployment workflow, from pushing a branch to deploying to production.
Sources: README.md:37-40, 48-51, CLAUDE.md:43-46
Conclusion
The build process for the Datadog documentation is a sophisticated system that combines the power of the Hugo static site generator with custom Node.js scripting to handle the unique challenges of maintaining a large-scale technical documentation platform. The automation of API reference page generation is a key feature, ensuring that documentation stays in sync with API specifications. The entire workflow is integrated into a modern CI/CD pipeline, enabling rapid and reliable updates to the public-facing documentation.
Sources: README.md:2, assets/scripts/build-reference-pages.js