Migration & Processing Scripts
Relevant source files
The following files were used as context for generating this wiki page:
Migration and processing scripts within the CircleCI Docs Static Site are specialized tools designed to facilitate the transition of content between formats, maintain internal link integrity, and automate the generation of technical documentation. These scripts serve as a bridge between legacy content structures and the modern Antora-based architecture, ensuring that metadata, cross-references, and API specifications are accurately processed during the build pipeline. Sources: ARCHITECTURE.md:27-31, TECHNICAL_REFERENCE.md:167-184
The scope of these scripts includes content transformation (converting YAML frontmatter to AsciiDoc attributes), link maintenance (updating cross-references), and complex API documentation pipelines that fetch, bundle, and patch OpenAPI specifications. Sources: TECHNICAL_REFERENCE.md:167-184, gulp.d/tasks/build-api-docs.js:100-115
Content Transformation and Maintenance
The project utilizes a suite of scripts to manage the lifecycle of AsciiDoc content and maintain architectural consistency across the documentation repository.
Frontmatter Conversion
The convert_frontmatter.py script is used to transform legacy YAML frontmatter—typically used in Jekyll-based sites—into native AsciiDoc attributes. This ensures that metadata like page titles, descriptions, and platform identifiers are correctly interpreted by the Antora static site generator. Sources: TECHNICAL_REFERENCE.md:168-170
Cross-Reference Management
Maintaining internal link integrity is handled by the update-xrefs.py script and the xref-mapping.json configuration. As content is moved or restructured within the Antora component architecture, this script identifies broken or legacy links and updates them to the correct xref:component:module:page.adoc format. Sources: TECHNICAL_REFERENCE.md:182-184, archive/scripts/xref-mapping.json:1-10
The following table summarizes the primary migration and maintenance scripts:
| Script Name | Purpose | Language |
|---|---|---|
convert_frontmatter.py | Converts YAML frontmatter to AsciiDoc attributes | Python |
update-xrefs.py | Updates cross-references across the documentation | Python |
convert_tabs.py | Converts HTML tabs to native AsciiDoc tabs extension | Python |
update_image_blocks.py | Updates image block syntax to follow Antora standards | Python |
gen-git-last-update-meta.sh | Generates metadata for the last update date from Git | Shell |
Sources: TECHNICAL_REFERENCE.md:168-182
API Documentation Processing Pipeline
The build-api-docs.js task defines a sophisticated processing pipeline for generating API documentation. Unlike standard content, API v2 documentation is dynamic and requires multiple processing stages before it can be integrated into the static site. Sources: gulp.d/tasks/build-api-docs.js:100-115
Processing Stages
The API processing logic follows a specific sequence to ensure data integrity and technical accuracy:
- Ingestion: Fetching live OpenAPI specs from the CircleCI API.
- Resolution: Bundling and dereferencing
$refpointers using Redocly to ensure all definitions are local. - Enrichment: Generating code samples (cURL, Node.js, Python, etc.) via Kong's httpsnippet.
- Customization: Applying JSON patches from
openapi-patch.jsonto override or add descriptions. - Validation: Linting the final spec to catch errors before HTML generation.
- Rendering: Building the final interactive HTML using Redocly.
Sources: gulp.d/tasks/build-api-docs.js:100-115, gulp.d/tasks/build-api-docs.js:189-195
Data Flow Diagram
The diagram below illustrates the flow of data through the API processing pipeline from source spec to the final build output.
The pipeline ensures that live API data is localized, enriched, and customized before being served. Sources: gulp.d/tasks/build-api-docs.js:115-185
Link Resolution and Mapping
The system relies on a centralized mapping file, xref-mapping.json, to resolve page aliases and legacy paths to their new component-based locations. Sources: archive/scripts/xref-mapping.json:1-20
Mapping Structure
Each entry in the mapping file identifies the target component, module, and filename, allowing scripts to reconstruct valid Antora cross-references. Sources: archive/scripts/xref-mapping.json:1-15
{
"managing-url-orbs-allow-lists": {
"component": "orbs",
"module": "use",
"filename": "managing-url-orbs-allow-lists.adoc"
},
"executor-intro": {
"component": "guides",
"module": "execution-managed",
"filename": "executor-intro.adoc"
}
}
Sources: archive/scripts/xref-mapping.json:2-6, archive/scripts/xref-mapping.json:71-75
Build Integration
Migration and processing scripts are integrated into the project's Gulp-based build system. While content migration scripts are typically run as manual maintenance tasks, the API processing script is a core part of the automated build pipeline. Sources: TECHNICAL_REFERENCE.md:167-168, gulp.d/tasks/build-api-docs.js:24-42
Execution Sequence
The build site task orchestrates the execution of these processing components:
Migration and processing scripts are fundamental to the maintenance of the CircleCI Docs Static Site. They automate the labor-intensive tasks of metadata conversion and link management while providing a robust pipeline for the continuous delivery of technical API documentation. By separating content transformation from the main site generation, the project maintains a clean source repository while delivering a highly integrated user experience. Sources: ARCHITECTURE.md:21-25, TECHNICAL_REFERENCE.md:188-202