[geo-optimizer] Add robots.txt with AI bot allow rules to docs site

[github-actions]

GEO Improvement: Add robots.txt with AI Bot Allow Rules to Docs Site

Source audit: GEO Audit Report — 2026-07-10
Audit date: 2026-07-10
Run: §29109438801

Finding

"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)"

The docs site at https://github.github.com/gh-aw/ scored 0/20 for the Robots.txt category — the single largest missing score block in the entire audit.

Why this matters

Without explicit allow rules in robots.txt, major AI crawlers (ChatGPT's GPTBot, Claude's ClaudeBot, Perplexity's PerplexityBot) may default to not indexing the docs site, directly preventing AI engines from citing gh-aw documentation in answers. Fixing this alone could recover up to 20 points on the docs site GEO score (currently 43/100).

Suggested fix

1. Add a robots.txt (or update the existing one) at the root of https://github.github.com/gh-aw/ with explicit allow rules:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: *
Allow: /

2. If the site is built with a static site generator (e.g., Docusaurus, Jekyll), add the robots.txt to the static/ or root assets directory so it is published at the correct path.
3. Verify the file is accessible at https://github.github.com/gh-aw/robots.txt after deployment.

Generated by 🌍 GEO Optimizer Daily Audit · 23 AIC · ⌖ 11.1 AIC · ⊞ 5K · ◷

• expires on Jul 17, 2026, 9:08 AM UTC-08:00