Repository navigation
Improve search engine / agent discoverability for DataFusion website #23258
Description
Is your feature request related to a problem or challenge?
As software development in general becomes more and more agent driven / search driven, it is important to make sure that the content on the DataFusion website is part of the content used to make those development decisions
If the datafusion website is invisible to agents then we won't show up when people ask said agents to help them build tools, etc
There are a few things that the https://datafusion.apache.org is clearly missing
/robots.txtwith clear crawl rules (basically should crawl everything) -- for example from duckdb: https://duckdb.org/robots.txt- /sitemap.xml listing canonical URLs, keep it updated on publish - for example from duckdb: https://duckdb.org/sitemap.xml
There is a bunch more stuff from https://isitagentready.com/datafusion.apache.org but I think robots.txt
Describe the solution you'd like
Add robots.txt and sitemap.xml
Ideally using one PR for each feature
The sitemap.xml should be auto generated as part of the sphinx build process
Describe alternatives you've considered
No response
Additional context
No response
Activity
aguilaredu commentedon Jun 30, 2026
take
aguilaredu commentedon Jun 30, 2026
Glad to help with this
alamb commentedon Jun 30, 2026
Thank you @aguilaredu 🙏
Please tag me on any PR you may have for reviews
imtherealnaska commentedon Aug 13, 2026
Hey can I pick this ? Would be my first issue . :-)
aguilaredu commentedon Aug 13, 2026
Hey @imtherealnaska, feel free to pick it up. Been really busy lately :). Thanks!
imtherealnaska commentedon Aug 13, 2026
take
feat: add sphinx-sitemap dependency and configuration for sitemap gen…
Jefffrey commentedon Aug 16, 2026
@alamb ill let you decide if want to add more things from that isitagentready site to this tracking issue, otherwise if we're fine with just sitemap and robots we can close this as resolved
feat: Add robots.txt to site (apache#24333)
alamb commentedon Aug 18, 2026
Let's close this one and we can open a new ticket if/when needed