Repository navigation
Adopt temporalio/snipsync for documentation #10768
Description
Activity
It looks like an interesting solution. We definitely need something like this, and I suppose I had stared building my own tooling, which is probably a bad idea.
The snipsync repo does not seem very active or popular and uses npm and yarn which I am not sure many DataFusion contributors would be familiar with.
I wonder if there are other solutions out there?
There are other such as https://github.com/SimonCropp/MarkdownSnippets, but I know Temporal is a popular OSS product, with excellent documentation, see https://temporal.io/blog/snipsync-overview
So I guess there are these two options, and maintaining our tool
I believe we now achieve this goal using the doctest! macro -- for example
datafusion/datafusion/core/src/lib.rs
Lines 618 to 643 in 37e54ee
Reacted by Edmondo Porcu@alamb there might still good reasons to consider snipsync or an approach such as the one that snipsync implements:
- We do have examples in two places: in the user guide and in
datafusion-examples. With snipsync apprach, they will all be in a single location and could be imported in the markdown - Using comments/tags, it would be possible to import in the markdown only a portion of the example by looking at comments, so that the reader can focus on the relevant part of the code
What do you think about modifying the macro to support these two features?
- external example (specifying the path)
- using comments in the rust code
use std::error::Error; // Snipstart: "Snip1" print!("foo"); // Snipend print!("Some uninteresting code"); // Snipstart: "Errors" panic!("Something went wrong"); // Snipend
You can print foo in Datafusion using the `print!` macro: <!--include examples/example_1.rs snip-name="Snip1" --> Another important feature in Datafusion is `panic!`, which can be used like so: <!--include examples/example_1.rs snip-name="Errors" -->
- We do have examples in two places: in the user guide and in
What do you think about modifying the macro to support these two features?
I think in theory it sounds great. I have no strong preference on tool or methodology.
What I think is important is that it is as easy as possible to write / maintain the documentation + examples (clear docs on how to document things, good examples of doing so with the docs)
I believe @ugoa 's changes in the following PR now test the examples in the documentation automatically so closing this issue
Is your feature request related to a problem or challenge?
Datafusion documentation is amazing and examples are too. However, inline snippets might fall out of sync from the codebase and are not very easy to consume because you often need to figure out the required imports to get a snippet to compile. On the other side, writing a full Rust script in a documentation page creates too much noise and make the documentation less effective.
Describe the solution you'd like
Ideally, we would have snippets that are portions of executable scripts, so that we can via CI/CD execute them and confirm they are not obsolete, but at the same time only the relevant section of the script is included in the docs.
We can use https://github.com/temporalio/snipsync for the purpose
Describe alternatives you've considered
No response
Additional context
I was working on a separate issue for documentation, and I have found myself facing the problem that I mentioned above, see #7306 (comment)