From ef2211dc67c97bc317272e2620d000df28861452 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 6 May 2026 04:46:36 +0000 Subject: [PATCH 1/3] Initial plan From 4875fa42cd2829b3781c83da9dbaae8f09cdf546 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 6 May 2026 04:57:57 +0000 Subject: [PATCH 2/3] docs: add robots.txt with sitemap directive for AI crawler discoverability Add docs/public/robots.txt with a Sitemap directive pointing to the sitemap-index.xml generated by @astrojs/sitemap. Also update root .gitignore to allow docs/public/robots.txt to be tracked (the *.txt rule was blocking it). Agent-Logs-Url: https://github.com/github/gh-aw/sessions/34744291-8995-4f84-9ca3-f034e20bcdcd Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com> --- .gitignore | 2 ++ docs/public/robots.txt | 4 ++++ 2 files changed, 6 insertions(+) create mode 100644 docs/public/robots.txt diff --git a/.gitignore b/.gitignore index 4df954b0975..d2c3ec9f325 100644 --- a/.gitignore +++ b/.gitignore @@ -166,8 +166,10 @@ test-inline-*.md test-inline-*.lock.yml *.txt *.zip +!docs/public/robots.txt # Temporary downloaded logs *.txt +!docs/public/robots.txt conclusion/ detection/ diff --git a/docs/public/robots.txt b/docs/public/robots.txt new file mode 100644 index 00000000000..07e706d02c1 --- /dev/null +++ b/docs/public/robots.txt @@ -0,0 +1,4 @@ +User-agent: * +Allow: / + +Sitemap: https://github.github.com/gh-aw/sitemap-index.xml From fba7ad434cb3225795f485154af4ec7ade863e51 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 6 May 2026 04:58:34 +0000 Subject: [PATCH 3/3] chore: remove duplicate robots.txt gitignore exception Agent-Logs-Url: https://github.com/github/gh-aw/sessions/34744291-8995-4f84-9ca3-f034e20bcdcd Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com> --- .gitignore | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitignore b/.gitignore index d2c3ec9f325..2b77997f03d 100644 --- a/.gitignore +++ b/.gitignore @@ -166,7 +166,6 @@ test-inline-*.md test-inline-*.lock.yml *.txt *.zip -!docs/public/robots.txt # Temporary downloaded logs *.txt !docs/public/robots.txt