Feat: generate llms.txt and markdown pages for AI agents
Description of your changes
Makes the documentation consumable by AI agents: generates an llms.txt index and a markdown copy of every docs page at build time.
docusaurus-plugin-llms(new dev-time plugin) emitsllms.txt,llms-full.txt, and a.mdtwin next to every page's HTML, so agents can fetch<page-url>.mdinstead of parsing themed HTML. Version output mirrors the site's routing: the rootllms.txtindexes the latest release served at/docs/, whilenextand older versions get their own indexes under/docs/<version>/llms.txt.scripts/llms-postprocess.js(runs via thepostbuildhook, so CI and Netlify-styleyarn buildboth trigger it) restructures the indexes into the llmstxt nested layout: CLI man-pages move to/docs/cli/llms.txtand other versions hang off/docs/versions/llms.txt. This keeps every index in the 77–91K character range — under the ~100K budget agents can ingest in one read — and the script fails the build if an index ever crosses it. It also prepends an index pointer to every generated markdown file.src/theme/Root.jsadds a visually hidden, server-rendered note on each page telling agents where the index and markdown variants live.
Measured with afdocs (the scanner behind CLOMonitor's agent readiness section) against a local production build: overall 94/100, with content discoverability at 92 and index coverage at 100%. The one category that still fails — Accept: text/markdown content negotiation — is a hosting limitation, not something the repo can fix.
I have:
- Read and followed KubeVela's contribution process.
- Update
sidebar.jsif adding a new page. - Run
yarn startto ensure the changes has taken effect.
Special notes for your reviewer
No docs pages or sidebar entries change — all new output is generated into build/ (verified with full yarn build runs rather than yarn start, since the plugin and postbuild hook only run on production builds). Build time grows by roughly 1–2 minutes for the markdown generation across versions. The commit is DCO signed off.
Summary by cubic
Makes the KubeVela docs consumable by AI agents: builds an llms.txt index and a markdown copy of every page at docs build time so agents can fetch https://kubevela.io/llms.txt or any page's .md twin instead of parsing styled HTML. Measured 94/100 on the CLOMonitor agent readiness scan (up from no index at all), with content discoverability 92 and index coverage 100%.
Build changes
- Adds the
docusaurus-plugin-llmsdependency and configures it to emit per-version indexes matching the site's routing (/docs/for the latest release,/docs/<version>/fornextand older versions). - Adds
scripts/llms-postprocess.jsas apostbuildhook that restructures the plugin output into the llmstxt nested layout: CLI man-pages move under<version>/cli/llms.txt, per-version indexes under/docs/versions/llms.txt, and every index stays under the ~100K character agent ingestion budget (the script fails the build if one exceeds it). - Chains the postprocessor into the standalone
Dockerfilebuild, which runsdocusaurus builddirectly and would otherwise skip yarn'spostbuildhook. - Prepends a pointer to the root
llms.txton every generated markdown page and adds a visually hidden note to every HTML page viasrc/theme/Root.js.
No docs pages or sidebar entries change; all output is generated into build/. Build time grows by roughly 1–2 minutes for markdown generation.