System Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
This document provides a technical overview of the Datadog Documentation repository's architecture. The system is a sophisticated static site generation pipeline designed to build and publish documentation to the Datadog documentation site. It leverages Hugo as its core static site generator, orchestrated by a combination of Makefiles, Node.js scripts, and Python tools.
The architecture is designed for scalability, supporting content in multiple languages, and integrating data from various external sources to generate comprehensive documentation. This includes fetching API client examples, integration metadata, and other content programmatically. The build and deployment process is highly automated, with strict contribution guidelines to ensure pipeline integrity and successful deployments.
Core Technologies and Components
The system is built upon a stack of open-source technologies, each serving a distinct purpose in the content generation and build process.
This diagram illustrates the primary components of the documentation system and their interactions.
Technology Stack
| Component | Technology | Purpose |
|---|---|---|
| Static Site Generator | Hugo | Compiles Markdown content and templates into a static website. It uses Goldmark for Markdown rendering. |
| Build Orchestration | Makefile | Top-level orchestrator for complex build processes, dependency installation, and development server management. |
| Scripting & Automation | Node.js (with Yarn) | Manages package dependencies, runs build automation scripts, and processes assets like SCSS and JavaScript. |
| Content Processing | Python | Used for various content-related tasks, including fetching integration data, processing translations, and pulling content from external sources. |
Sources: README.md:1-3, 37-38, CLAUDE.md:5-8, 20-27
Build System and Orchestration
The build system is a multi-layered process orchestrated by Makefile and executed via Node.js and Python scripts. It is responsible for fetching dependencies, processing content, and running the Hugo build.
Sources: CLAUDE.md:10-27
Build Commands
A series of make and yarn commands are available to manage the development and build lifecycle.
| Command | Description |
|---|---|
make start / yarn start | Performs a full build, including fetching external dependencies, and runs the development server. |
make build / yarn build | Executes a production build of the static site. |
make dependencies | Installs all project dependencies, including Node.js, Python, and external content repositories. |
make clean-all | Cleans the entire environment, including generated files and fetched repositories. |
Sources: CLAUDE.md:11-27
Build Process Flow
The following diagram shows the sequence of operations during a typical local development build.
Sources: CLAUDE.md:11-15, 24-27, 31-38
Modular Architecture Concept
The system's provisioning philosophy can be broken down into a modular file/package/service model, which is a standard for enterprise software deployment.
- File(s): Configuration files (
conf.yaml.example,manifest.json) stored under source control. - Package: Binaries and their deployment, such as containerized Agent packages or Python wheels, hosted in internal repositories.
- Service: Manages the runtime instance, for example, the Datadog Agent service on a host.
This structure allows for robust management using Infrastructure-as-Code (IaC) tools like Ansible, Terraform, and Chef.
Sources: content/en/administrators_guide/build.md:60-78
Content and Data Architecture
The repository's primary purpose is to manage documentation content. This content is structured in a way that facilitates both manual authoring and automatic generation.
Sources: CLAUDE.md:29-38
Directory Structure
The content and its presentation are organized into several key directories.
| Directory | Purpose |
|---|---|
content/en/ | Contains the primary English Markdown documentation files. |
content/{fr,es,ja,ko}/ | Contains translated content, which is managed externally and not edited directly in the repository. |
layouts/ | Hugo templates, including shortcodes used for rendering complex components like multi-language code blocks and integration assets. |
assets/ | SCSS stylesheets and JavaScript files that are processed during the build. |
static/ | Static assets like images and fonts that are copied directly to the output directory. |
config/ | Hugo configuration files for different environments (development, preview, live). |
Sources: CLAUDE.md:30-38, 43-45
Integration Data Model
Integrations are a core part of the documentation, and their assets are defined by a strict schema. The manifest.json file is central to this model.
manifest.json Structure
This file defines metadata for an integration tile on the Datadog website.
{
"manifest_version": "2.0.0",
"display_on_public_website": true,
"tile": {
"title": "My Integration",
"media": [],
"overview": "A short overview."
},
"author": {
"name": "My Company"
},
"assets": {
"dashboards": {},
"monitors": {},
"service_checks": "assets/service_checks.json"
},
"classifier_tags": [
"Category::Cloud",
"Supported OS::Linux",
"Submitted Data Type::Metrics"
]
}
Sources: content/en/developers/integrations/check_references.md:52-60
Software Catalog Entity Model
The documentation also defines the architecture for Datadog's Internal Developer Portal (IDP), which uses a system of entities. These entities represent the building blocks of a microservice architecture.
The diagram below shows the relationships between native entity types in the Software Catalog. A system can be composed of multiple other entities, including services, APIs, and even other systems.
An example of a system entity definition:
apiVersion: v3
kind: system
metadata:
name: product-recommendation
displayName: "Product Recommendation"
owner: shopist
spec:
lifecycle: production
components:
- service:product-recommendation
- service:orders-app
- api:products
- system:shopist-user-trends
Sources: content/en/internal_developer_portal/software_catalog/entity_model/native_entities.md:46-61
Deployment Pipeline
The deployment process is fully automated and triggered by merges to the master branch. A strict branch naming convention is enforced to enable CI/CD functionality, including the generation of preview environments.
Sources: README.md:26-36, 44-46, CONTRIBUTING.md:10-11
Branch Naming Convention
All branches created by Datadog employees MUST follow the <name>/<description> format. This is critical because the GitLab pipeline relies on the forward slash (/) to trigger builds and generate branch previews. Failure to adhere to this convention results in a failed CI check.
Sources: README.md:28-30, CLAUDE.md:47-50
Deployment Flow
The sequence diagram below outlines the process from a developer pushing a change to its deployment in production.
Sources: README.md:26-46, CLAUDE.md:47-53
Conclusion
The Datadog Documentation repository is a robust, highly-automated system for managing and publishing technical documentation at scale. Its architecture combines a powerful static site generator (Hugo) with a flexible orchestration layer (Make, Node.js, Python) to handle complex requirements like external data fetching and asset processing. The integrated CI/CD pipeline, governed by strict contribution rules, ensures that content is reviewed, previewed, and deployed efficiently and reliably. This well-defined architecture allows contributors to focus on creating high-quality content while the system handles the complexities of the build and release process.
Sources: README.md, CLAUDE.md, content/en/administrators_guide/build.md