Include the latest rustdocs (e.g. cargo doc output on https://arrow.apache.org/datafusion/) #5981

Is your feature request related to a problem or challenge?

As part of consolidating our documentation and making it better we have been getting the lower level technical documentation about DataFusion into the rustdocs as explained here:

https://arrow.apache.org/datafusion/contributor-guide/architecture.html#architecture

This is good in that keeps the docs close to the code, but it is non ideal in that the docs on crates.io https://docs.rs/datafusion/latest/datafusion/index.html will only reflect released versions and thus will lag behind the latest docs on github

Describe the solution you'd like

I think it is a great idea -- thank you @waynexia

I actually think if we could build those API docs as part of the https://github.com/apache/arrow-datafusion/blob/main/docs build, they would "automatically" get hosted on https://arrow.apache.org/datafusion/

So the idea is if we could get

cd docs
make html

to produce the rustdocs somwhere and link them in somewhere that would be great

More details

https://arrow.apache.org/datafusion/ is published via some ASF mechanism that is similar to github pages

Specifically, this workflow

https://github.com/apache/arrow-datafusion/blob/388f9ec3e7f7c09dac56ee0fe074ca97a6af9d44/.github/workflows/docs.yaml#L12-L64

pushes to the https://github.com/apache/arrow-datafusion/tree/asf-site branch which then gets hosted via this magic yaml:

https://github.com/apache/arrow-datafusion/blob/388f9ec3e7f7c09dac56ee0fe074ca97a6af9d44/.asf.yaml#L48-L52

Describe alternatives you've considered

No response

Additional context

Suggested by @waynexia #5962 (comment)