Managing API Endpoints

Relevant source files

The following files were used as context for generating this wiki page:

• assets/scripts/build-api-pages.js
• assets/scripts/build-reference-pages.js
• content/en/api/README.md
• content/en/internal_developer_portal/software_catalog/endpoints/_index.md
• content/en/api/v2/users/_index.md
• layouts/partials/api/get-endpoint.html
• content/en/api/v2/users/request.CreateUser.json
• content/en/developers/integrations/check_references.md

The management of API endpoint documentation in this project is a fully automated, spec-driven process. Instead of manually writing markdown for each endpoint, the documentation is generated directly from OpenAPI specification files. This ensures that the API documentation is always in sync with the API definition, providing accuracy and consistency.

The core of this system is a set of Node.js scripts that parse the API specifications, generate content files (Markdown and JSON), and configure the site navigation. These generated assets are then used by Hugo layouts to render the final, user-facing API documentation pages. This approach separates the API definition (the source of truth) from its presentation.

Sources: content/en/api/README.md, assets/scripts/build-api-pages.js

API Documentation Generation Pipeline

The generation of API documentation follows a clear, automated pipeline orchestrated by build scripts. The process begins with OpenAPI specification files and ends with fully rendered HTML pages.

This diagram illustrates the high-level data flow for generating the API documentation.

Source of Truth: OpenAPI Specifications

All API documentation is derived from OpenAPI specification files, typically named full_spec.yaml for each API version (v1, v2). These files are the single source of truth. Any changes to API endpoints, parameters, or descriptions must be made in these specification files, not in the generated markdown content. The build process reads these files to begin the documentation generation.

Sources: content/en/api/README.md:3-4, assets/scripts/build-api-pages.js:654-679

Build Scripts

The primary scripts responsible for this process are build-api-pages.js and build-reference-pages.js. These scripts contain the logic for parsing specs and generating all necessary files for Hugo.

The main function processSpecs orchestrates the entire workflow. It iterates through each provided spec file, dereferences JSON schema references using @apidevtools/json-schema-ref-parser, and then calls a series of helper functions to generate the various assets.

Key functions within the build scripts include:

Function	Description	Source File
processSpecs	Main function that orchestrates the processing of an array of spec files.	assets/scripts/build-api-pages.js:630
createPages	Creates the directory structure and _index.md files for each API tag.	assets/scripts/build-api-pages.js:125
createResources	Generates examples.json files containing request/response examples and HTML schema tables.	assets/scripts/build-api-pages.js:156
updateMenu	Updates the menus/api.en.yaml file to build the API navigation sidebar.	assets/scripts/build-api-pages.js:21
filterExampleJson	Recursively filters the schema to generate JSON examples for requests and responses.	assets/scripts/build-api-pages.js:463
schemaTable	Converts a JSON schema object into an HTML table for displaying attributes.	assets/scripts/build-api-pages.js:582
createTranslations	Generates JSON files with strings for translation.	assets/scripts/build-api-pages.js:606

Sources: assets/scripts/build-api-pages.js, assets/scripts/build-reference-pages.js

Generated Content and Resources

The build scripts produce several types of files that are consumed by the Hugo static site generator.

Directory and Page Structure

For each tag defined in the OpenAPI spec (e.g., "Users", "Metrics"), the createPages function generates a corresponding directory and a headless _index.md file. This structures the content logically by API resource.

Example of a generated _index.md file for the "Users" API section:

---
title: Users
headless: true
---

This simple file serves as a container for the endpoints that will be rendered within it.

Sources: assets/scripts/build-api-pages.js:125-148, content/en/api/v2/users/_index.md

JSON Resource Files

The createResources function generates an examples.json file within each API section's directory. This file is crucial as it contains pre-processed data for rendering, including:

• Request body examples in JSON format.
• Example JSON for cURL requests (json_curl).
• Pre-rendered HTML tables for request and response schemas.
• Example JSON for various response codes (e.g., 200, 403).

This pre-generation optimizes the build process by offloading the complex schema-to-HTML conversion from Hugo's templating engine to the Node.js build script.

The following diagram shows the structure of the generated examples.json file for a single operation ID.

Schema-to-HTML Conversion

The schemaTable and rowRecursive functions in the build scripts are responsible for walking through the JSON schema of a request or response and generating a detailed HTML table. This table displays each field's name, type, description, and whether it is required. This is a key feature for making the API documentation readable and easy to use.

Sources: assets/scripts/build-api-pages.js:582-604, [assets/scripts/build-reference-pages.js:685-768]

Rendering with Hugo

The final step is rendering the generated content into HTML. Hugo uses partial templates to display the endpoint information.

The layouts/partials/api/get-endpoint.html partial is a key component in this stage. It is responsible for rendering a single API endpoint block. It takes the data from the dereferenced spec and the generated examples.json file to construct the final view, which includes:

• The endpoint's title, HTTP method, and path.
• A description of the endpoint.
• Tabs for different programming languages to show request examples.
• The request and response schema tables generated by the build script.

Sources: layouts/partials/api/get-endpoint.html

Endpoint Management in the Software Catalog

Beyond documentation generation, API endpoints are also managed as observable entities within the Datadog Internal Developer Portal's Software Catalog. This provides a different, operational view of the APIs.

Key features include:

• Automatic Discovery: Endpoints are automatically discovered and populated in the Endpoints list if the services are monitored with Datadog APM.
• Centralized View: The catalog consolidates performance, reliability, and ownership data for all APIs.
• Performance Monitoring: Users can track metrics like latency, request rate, and error rates for each endpoint.
• Ownership and Governance: Teams can be assigned as owners to endpoints, streamlining communication and accountability.

This system allows developers and managers to not only understand how to use an API but also how it is performing in a live environment.

Sources: content/en/internal_developer_portal/software_catalog/endpoints/_index.md

Summary

Managing API endpoints in this repository is a robust, automated system that leverages OpenAPI specifications as a single source of truth. Through a pipeline of build scripts, content and resource files are generated, which are then rendered by Hugo into a comprehensive and consistent API documentation site. This spec-driven approach ensures documentation accuracy and significantly reduces manual effort. Additionally, the integration with the Software Catalog provides a powerful operational view, allowing teams to monitor and govern their APIs effectively.