Processing llms.txt

Relevant source files

The following files were used as context for generating this wiki page:

• sync.ts
• README.md
• skills/claude-code-docs/SKILL.md
• hooks/rewrite-docs-url.sh
• hooks/hooks.json
• hooks/rewrite-docs-url.test.sh

The llms.txt file serves as the central manifest and source of truth for mirroring Claude Code documentation. Within the claude-code-docs project, this manifest is used to track available documentation pages, manage deletions, and facilitate efficient retrieval of markdown content for AI agents.

The project employs a synchronization script to parse llms.txt, download the referenced markdown pages, and organize them into a structured docs/ directory. This process ensures that the mirror remains an accurate, versioned representation of the canonical documentation located at code.claude.com/docs.

Sources: README.md:1-12, sync.ts:1-5

Sync Architecture and Data Flow

The synchronization logic is primarily contained in sync.ts. It follows a structured workflow to ensure data integrity and atomicity of the documentation mirror.

Synchronization Workflow

1. Manifest Retrieval: The script fetches the primary llms.txt manifest and the bundled llms-full.txt from the canonical source.
2. Validation: A safety check ensures that the manifest contains a minimum number of URLs (currently 10) to prevent accidental mass deletions due to a broken manifest parse.
3. Cleanup: The existing docs/ directory is recursively removed to ensure a clean state.
4. Parallel Fetching: Individual markdown pages are fetched in parallel using a configurable concurrency limit.
5. Link Rewriting: As pages are downloaded, internal links are rewritten to support local file navigation within the mirror.

The following diagram illustrates the data flow from the canonical documentation source to the local mirror:

Key Synchronization Components

Component	Description	Reference
BASE	The root URL for canonical documentation (https://code.claude.com/docs).	sync.ts:7
CONCURRENCY	Maximum number of simultaneous network requests (set to 10).	sync.ts:8
RETRIES	Number of attempts for failed network requests (set to 3).	sync.ts:9
rewriteLinks	Function that converts site-absolute links to relative markdown file paths.	sync.ts:17-40
fetchText	Utility function for HTTP GET requests with built-in retry logic.	sync.ts:42-56

Manifest Parsing and Content Extraction

The system identifies documentation pages by applying a regular expression to the contents of llms.txt.

URL Extraction Logic

The script extracts URLs that match the pattern: https://code.claude.com/docs/en/[^)\s]+\.md. This specifically targets the English locale markdown files listed in the manifest.

const urls = [
  ...new Set(
    [...manifest.matchAll(/https:\/\/code\.claude\.com\/docs\/en\/[^)\s]+\.md/g)].map((m) => m[0]),
  ),
];

Sources: sync.ts:60-64

Slug Management

To facilitate link rewriting, the system maintains a knownSlugs set. A "slug" is defined as the path segment of the URL between the locale and the file extension (e.g., en/agents.md has the slug agents). This set allows the rewriteLinks function to distinguish between internal pages that exist in the mirror and external references.

Sources: sync.ts:72-74

Link Transformation Logic

Since the canonical documentation uses site-absolute paths (e.g., /en/sub-agents), they would break when viewed as local files on platforms like GitHub. The rewriteLinks function transforms these into relative paths.

Rewrite Rules

1. Internal Links: If a link targets a slug found in knownSlugs, it is converted to a relative path ending in .md (e.g., ../sub-agents.md).
2. Anchor Normalization: Percent-encoded characters in fragments (like %2F) are stripped to match GitHub's generated heading ID format.
3. External Fallback: Links to pages not present in the manifest are converted to absolute URLs pointing back to the canonical site.

Integration with Tooling

The processed llms.txt and the resulting docs/ structure are utilized by external tools to provide documentation access to AI agents.

Agent Skills and Search

Tools like ask and csp (Code Search Please) use the mirror for semantic search.

• Discovery: The llms.txt file at the root of the mirror provides a fast way to shortlist pages via its one-line summaries.
• Search Scoping: AI agents are instructed to search the docs/ directory specifically to avoid duplicate results from the 5MB llms-full.txt bundle.

Sources: README.md:37-45, skills/claude-code-docs/SKILL.md:33-40

URL Rewriting Hooks

The project includes a PreToolUse hook for the Claude Code plugin. This hook monitors WebFetch calls and intercepts requests to code.claude.com/docs. It uses logic similar to the sync script to ensure that requests for HTML pages are redirected to the clean markdown versions processed during synchronization.

Input URL	Rewritten URL
.../docs/en/agents	.../docs/en/agents.md
.../docs/en/hooks#anchor	.../docs/en/hooks.md

Sources: hooks/rewrite-docs-url.sh:1-40, README.md:52-60

Conclusion

Processing llms.txt is the foundational step for maintaining the claude-code-docs mirror. By treating the manifest as the source of truth, the system automates the lifecycle of documentation pages—including additions and deletions—while optimizing the content for both human readability on GitHub and programmatic access by AI coding agents. Through parallelized fetching and intelligent link rewriting, the project transforms a centralized web resource into a portable, searchable, and developer-friendly documentation repository.