Is your feature request related to a problem? Please describe.
#3285 shipped our first llms.txt at designsystem.digital.gov/llms.txt, closing #3284, and it got us a first-party discovery surface at a point where a lot of teams are using agents against our docs. This issue captures the follow-up work we identified while reviewing it — two enhancements that build on what's now in place.
1. Extend the generated sections to the rest of the file. Two lists in pages/llms.txt already generate themselves: components (from the same query as the components overview page) and utilities (from the utilities collection). That's the pattern we want everywhere. The remaining ~54 link lines — getting started, settings, design tokens, accessibility, templates, patterns, code and design assets, and everything under ## Optional — are still curated by hand, along with each link's one-line description, which means keeping them accurate depends on someone remembering this file exists when they rename or retire a page. Deriving those lists from what the site publishes removes that dependency.
2. Serve agents markdown instead of browser HTML. Links in the file currently resolve to full pages: nav, sidebar, in-page navigation, accordions, script tags, search widget, footer. _site/components/button/index.html is ~121 KB of HTML wrapping ~74 KB of <main>, and only a fraction of that is guidance. An agent spends context stripping our chrome before it reads a word of our content, on every request.
The convention anticipates this: llmstxt.org expects links to resolve to clean markdown. Nord Health's design system is a useful reference — they serve markdown files directly rather than pointing agents at the same HTML a browser gets.
Describe the solution you'd like
Two related changes, either of which stands alone, and which are better together.
A. Generate llms.txt from the site's own page inventory
Build the remaining sections from the pages Jekyll actually publishes, using each page's existing front matter (title, lead, category) for its label and description. Add a page to the site and it appears; move a page and its link moves with it; retire a page and its entry disappears.
This extends what the component and utility lists already do to the rest of the file.
B. Publish a markdown companion for every documentation page
Emit a markdown version of each page's main content alongside the HTML — /components/button/index.md, or whichever URL shape we settle on — and point the links in llms.txt at those instead of at the HTML. Content only, no nav, no sidebar, no scripts.
Once markdown companions exist, llms-full.txt becomes a concatenation of files we already generate rather than a separate maintenance burden (see the size constraint below before committing to it).
Constraints and open questions
These came out of reading the current build. They're the parts most likely to bite an implementer, so they're worth deciding before code gets written.
Generation has to happen inside Jekyll, not after it. cloud.gov Pages runs npm run federalist (which is npx gulp build) and then builds Jekyll itself. There is no post-jekyll build step in the deploy pipeline. A Node post-processor over _site/ would work locally and in CircleCI and then silently never run in production — the worst failure shape available, since the file would exist and just be stale. A generator or hook in _plugins/ (where uswds_version.rb, library_component.rb, and the others already live) runs in every environment. Worth confirming against the current Pages build config before building on it.
Component pages have no markdown body to serve. This is the one that breaks the obvious approach. _components/button/button.md is front matter and nothing else — same for accordion, combo box, and the rest. The prose lives in _components/<name>/guidance/*.md partials, and the settings and variants tables are generated by _includes/component-guidance.html from _data/settings/. So "serve the source markdown" produces empty files for the 47 pages an agent most wants to read. Markdown has to come either from rendered output (a post_render/post_write hook converting <main>, needing an HTML-to-markdown converter — reverse_markdown is the usual Ruby choice and would be a new gem) or from a markdown-emitting layout that re-includes the guidance partials without the HTML scaffolding. Worth prototyping one component page both ways before picking.
The sitemap is a starting point, not the list. sitemap.xml has 257 URLs; roughly 88 of them probably don't belong in an agent index — 44 per-component accessibility-tests/ stubs, 25 whats-new/ posts, 14 next/ report pages, 3 PDFs. Whatever we generate from needs an explicit include/exclude policy, and that policy should live in one place rather than being spread across llms.txt, .pa11yci, and the proof script the way our current exclusions are.
<loc> values are root-relative. _config.yml sets no url:, so the sitemap emits /components/button/ rather than a full URL. A sitemap-driven generator inherits the same problem #3285 solved by defining the domain once at the top of the Liquid. Setting url: site-wide would change the SEO tags and sitemap output everywhere, which is a bigger decision than this issue should make on its own — so either keep the single constant or open that as its own change.
llms-full.txt needs a size policy before it needs an implementation. Main-content text across the 269 built pages is about 3.2 MB, and /components/icon/ is 582 KB of that by itself — it renders ~243 icons and already gets excluded from pa11y for exceeding a 120s timeout. Concatenating everything produces a file too large for most context windows to be useful. Decide what's in scope (components and documentation, plausibly; not posts) and whether anything gets truncated.
Not every page has a lead. Ten pages across pages/ and the collections have no lead in front matter (pages/404.md, pages/all.html, pages/home.md, pages/documentation/code-guidelines.md, and others). Generated descriptions need a defined fallback, or those pages need a lead added, or they belong on the exclude list. Any of the three is fine; leaving it undefined produces entries with dangling colons.
Markdown files stay out of the sitemap for free. jekyll-sitemap only includes .htm, .html, .xhtml, and .pdf, so .md output won't land in sitemap.xml and pa11y won't try to audit plain text. That's the behavior we want, and it's worth a test pinning it so a future gem upgrade can't quietly change it.
Check how cloud.gov Pages serves .md. The companion files are only useful if a GET returns them as readable text rather than triggering a download. Verify Content-Type on a preview build before this is called done.
spec/llms_txt_spec.rb is a downstream consumer, not just a test file. Its assertions are built around the curated lists and around a regex matching the uswdsComponents assign. Generating the file invalidates most of them. They should be rewritten to assert the generated contract — shape, that every emitted link resolves to something the build wrote, that markdown companions exist for every linked page — rather than patched to keep passing.
Acceptance criteria
llms.txt is generated from the site's published page inventory; adding, moving, or retiring a documentation page updates it with no hand edit- Include/exclude rules for which pages appear are defined in one place, with the reasoning recorded
- Every documentation page in scope has a markdown companion containing its main content and none of the page chrome
- Links in
llms.txt resolve to the markdown companions
GET on llms.txt and on any companion returns 200 with a content type a client reads as text- Markdown companions stay out of
sitemap.xml, and a test holds that
- Generation runs in the cloud.gov Pages build, verified on a preview — not only locally and in CircleCI
spec/llms_txt_spec.rb is rewritten against the generated contract, and a moved or deleted page surfaces as a failing test rather than a live 404bundle exec rspec, npm run lint, and the pa11y jobs pass
Additional context
Follows up #3285 (merged) and #3284 (closed).
Suggested sequencing: A before B. Once the index is generated, repointing links at markdown is a one-expression change; the other order means hand-editing ~54 lines and then discarding that work.
Reference implementation to compare against: https://nordhealth.design/llms.txt
Convention: https://llmstxt.org/
Review context: #3285 (review)
Code of Conduct
Is your feature request related to a problem? Please describe.
#3285 shipped our first
llms.txtat designsystem.digital.gov/llms.txt, closing #3284, and it got us a first-party discovery surface at a point where a lot of teams are using agents against our docs. This issue captures the follow-up work we identified while reviewing it — two enhancements that build on what's now in place.1. Extend the generated sections to the rest of the file. Two lists in
pages/llms.txtalready generate themselves: components (from the same query as the components overview page) and utilities (from the utilities collection). That's the pattern we want everywhere. The remaining ~54 link lines — getting started, settings, design tokens, accessibility, templates, patterns, code and design assets, and everything under## Optional— are still curated by hand, along with each link's one-line description, which means keeping them accurate depends on someone remembering this file exists when they rename or retire a page. Deriving those lists from what the site publishes removes that dependency.2. Serve agents markdown instead of browser HTML. Links in the file currently resolve to full pages: nav, sidebar, in-page navigation, accordions, script tags, search widget, footer.
_site/components/button/index.htmlis ~121 KB of HTML wrapping ~74 KB of<main>, and only a fraction of that is guidance. An agent spends context stripping our chrome before it reads a word of our content, on every request.The convention anticipates this: llmstxt.org expects links to resolve to clean markdown. Nord Health's design system is a useful reference — they serve markdown files directly rather than pointing agents at the same HTML a browser gets.
Describe the solution you'd like
Two related changes, either of which stands alone, and which are better together.
A. Generate
llms.txtfrom the site's own page inventoryBuild the remaining sections from the pages Jekyll actually publishes, using each page's existing front matter (
title,lead,category) for its label and description. Add a page to the site and it appears; move a page and its link moves with it; retire a page and its entry disappears.This extends what the component and utility lists already do to the rest of the file.
B. Publish a markdown companion for every documentation page
Emit a markdown version of each page's main content alongside the HTML —
/components/button/index.md, or whichever URL shape we settle on — and point the links inllms.txtat those instead of at the HTML. Content only, no nav, no sidebar, no scripts.Once markdown companions exist,
llms-full.txtbecomes a concatenation of files we already generate rather than a separate maintenance burden (see the size constraint below before committing to it).Constraints and open questions
These came out of reading the current build. They're the parts most likely to bite an implementer, so they're worth deciding before code gets written.
Generation has to happen inside Jekyll, not after it. cloud.gov Pages runs
npm run federalist(which isnpx gulp build) and then builds Jekyll itself. There is no post-jekyll buildstep in the deploy pipeline. A Node post-processor over_site/would work locally and in CircleCI and then silently never run in production — the worst failure shape available, since the file would exist and just be stale. A generator or hook in_plugins/(whereuswds_version.rb,library_component.rb, and the others already live) runs in every environment. Worth confirming against the current Pages build config before building on it.Component pages have no markdown body to serve. This is the one that breaks the obvious approach.
_components/button/button.mdis front matter and nothing else — same for accordion, combo box, and the rest. The prose lives in_components/<name>/guidance/*.mdpartials, and the settings and variants tables are generated by_includes/component-guidance.htmlfrom_data/settings/. So "serve the source markdown" produces empty files for the 47 pages an agent most wants to read. Markdown has to come either from rendered output (apost_render/post_writehook converting<main>, needing an HTML-to-markdown converter —reverse_markdownis the usual Ruby choice and would be a new gem) or from a markdown-emitting layout that re-includes the guidance partials without the HTML scaffolding. Worth prototyping one component page both ways before picking.The sitemap is a starting point, not the list.
sitemap.xmlhas 257 URLs; roughly 88 of them probably don't belong in an agent index — 44 per-componentaccessibility-tests/stubs, 25whats-new/posts, 14next/report pages, 3 PDFs. Whatever we generate from needs an explicit include/exclude policy, and that policy should live in one place rather than being spread acrossllms.txt,.pa11yci, and theproofscript the way our current exclusions are.<loc>values are root-relative._config.ymlsets nourl:, so the sitemap emits/components/button/rather than a full URL. A sitemap-driven generator inherits the same problem #3285 solved by defining the domain once at the top of the Liquid. Settingurl:site-wide would change the SEO tags and sitemap output everywhere, which is a bigger decision than this issue should make on its own — so either keep the single constant or open that as its own change.llms-full.txtneeds a size policy before it needs an implementation. Main-content text across the 269 built pages is about 3.2 MB, and/components/icon/is 582 KB of that by itself — it renders ~243 icons and already gets excluded from pa11y for exceeding a 120s timeout. Concatenating everything produces a file too large for most context windows to be useful. Decide what's in scope (components and documentation, plausibly; not posts) and whether anything gets truncated.Not every page has a
lead. Ten pages acrosspages/and the collections have noleadin front matter (pages/404.md,pages/all.html,pages/home.md,pages/documentation/code-guidelines.md, and others). Generated descriptions need a defined fallback, or those pages need aleadadded, or they belong on the exclude list. Any of the three is fine; leaving it undefined produces entries with dangling colons.Markdown files stay out of the sitemap for free. jekyll-sitemap only includes
.htm,.html,.xhtml, and.pdf, so.mdoutput won't land insitemap.xmland pa11y won't try to audit plain text. That's the behavior we want, and it's worth a test pinning it so a future gem upgrade can't quietly change it.Check how cloud.gov Pages serves
.md. The companion files are only useful if aGETreturns them as readable text rather than triggering a download. VerifyContent-Typeon a preview build before this is called done.spec/llms_txt_spec.rbis a downstream consumer, not just a test file. Its assertions are built around the curated lists and around a regex matching theuswdsComponentsassign. Generating the file invalidates most of them. They should be rewritten to assert the generated contract — shape, that every emitted link resolves to something the build wrote, that markdown companions exist for every linked page — rather than patched to keep passing.Acceptance criteria
llms.txtis generated from the site's published page inventory; adding, moving, or retiring a documentation page updates it with no hand editllms.txtresolve to the markdown companionsGETonllms.txtand on any companion returns 200 with a content type a client reads as textsitemap.xml, and a test holds thatspec/llms_txt_spec.rbis rewritten against the generated contract, and a moved or deleted page surfaces as a failing test rather than a live 404bundle exec rspec,npm run lint, and the pa11y jobs passAdditional context
Follows up #3285 (merged) and #3284 (closed).
Suggested sequencing: A before B. Once the index is generated, repointing links at markdown is a one-expression change; the other order means hand-editing ~54 lines and then discarding that work.
Reference implementation to compare against: https://nordhealth.design/llms.txt
Convention: https://llmstxt.org/
Review context: #3285 (review)
Code of Conduct