Skip to content

docs: add robots.txt with sitemap directive to fix AI crawler discoverability - #30503

Merged
pelikhan merged 3 commits into
mainfrom
copilot/fix-sitemap-generation-docs-site
May 6, 2026
Merged

pelikhan merged 3 commits into
mainfrom
copilot/fix-sitemap-generation-docs-site

Conversation

Copilot AI commented May 6, 2026 •

Copy link
Copy Markdown
Contributor

https://github.github.com/gh-aw/sitemap.xml was returning 404, blocking AI crawlers from discovering interior pages. The @astrojs/sitemap integration was already correctly wired up in astro.config.mjs and package.json — the missing piece was a robots.txt to surface the sitemap location to crawlers.

Changes

  • docs/public/robots.txt (new): Permits all crawlers and declares Sitemap: https://github.github.com/gh-aw/sitemap-index.xml
  • .gitignore: Added !docs/public/robots.txt exception after the *.txt wildcard rule that was silently preventing the file from being tracked
User-agent: *
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap-index.xml

Note: @astrojs/sitemap generates sitemap-index.xml (not sitemap.xml). The <link rel="sitemap"> tag is injected into all page heads automatically by the integration at build time.

Copilot AI linked an issue May 6, 2026 that may be closed by this pull request
1 of 4 tasks
Copilot AI and others added 2 commits May 6, 2026 04:57
…ility

Add docs/public/robots.txt with a Sitemap directive pointing to the
sitemap-index.xml generated by @astrojs/sitemap. Also update root
.gitignore to allow docs/public/robots.txt to be tracked (the *.txt
rule was blocking it).

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/34744291-8995-4f84-9ca3-f034e20bcdcd

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Agent-Logs-Url: https://github.com/github/gh-aw/sessions/34744291-8995-4f84-9ca3-f034e20bcdcd

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix sitemap generation on docs site docs: add robots.txt with sitemap directive to fix AI crawler discoverability May 6, 2026
Copilot AI requested a review from gh-aw-bot May 6, 2026 04:59
@pelikhan
pelikhan marked this pull request as ready for review May 6, 2026 05:05
Copilot AI review requested due to automatic review settings May 6, 2026 05:05
@pelikhan
pelikhan merged commit 6f5e2a9 into main May 6, 2026
@pelikhan
pelikhan deleted the copilot/fix-sitemap-generation-docs-site branch May 6, 2026 05:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a robots.txt file for the docs site so crawlers can discover the generated sitemap (notably sitemap-index.xml), and updates .gitignore to ensure the new file is tracked despite existing *.txt ignore rules.

Changes:

  • Added docs/public/robots.txt that allows all crawlers and declares the sitemap URL.
  • Updated .gitignore to un-ignore docs/public/robots.txt so it’s committed.
Show a summary per file
File Description
docs/public/robots.txt Introduces a robots file that declares the sitemap location for crawler discovery.
.gitignore Adds an exception so docs/public/robots.txt is not excluded by *.txt ignore rules.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 1/2 changed files
  • Comments generated: 1

Comment thread docs/public/robots.txt
Comment on lines +3 to +4

Sitemap: https://github.github.com/gh-aw/sitemap-index.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[plan] Fix sitemap generation on docs site (Astro Starlight)

4 participants