Skip to content

Library Guide: Building LogicalPlans #7306

Description

@alamb

Is your feature request related to a problem or challenge?

Part of #7014

If we want to have DataFusion used as the core of many new systems, we need it to be as easy as possible for someone to get their idea working on top of DataFusion.

Thanks to @tshauck we now have a basic Library Users Guide ❤️ and this ticket describes expanding it out

Describe the solution you'd like

Fill in the content of https://arrow.apache.org/datafusion/library-user-guide/building-logical-plans.html

Example Outline

  1. Building LogicalPlans directly
  2. Building LogicalPlans using LogicalPlanBuilder
  3. Translating LogicalPlan --> ExecutionPlan via physical planner

The examples directory holds a bunch more of examples: https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. self-assigned this
    on Oct 26, 2023
  2. removed their assignment
    on May 23, 2024
  3. edmondop commented on May 24, 2024

    @edmondop
    Contributor

    @alamb I checked the previous @andygrove PR and it seems like we have a good setup. I read the docs, and noticed they lack a paragraph on how to translate the LogicalPlan into Physical Plan .

    What do you think we should do? I can add that paragraph, but maybe we also want to link the examples? What else do you think it's needed to close this

  4. alamb commented on May 25, 2024

    @alamb
    ContributorAuthor

    What do you think we should do? I can add that paragraph, but maybe we also want to link the examples? What else do you think it's needed to close this

    Those both sound like excellent ideas to close off this issue. Thank you @edmondop

  5. edmondop commented on May 25, 2024

    @edmondop
    Contributor

    What do you think we should do? I can add that paragraph, but maybe we also want to link the examples? What else do you think it's needed to close this

    Those both sound like excellent ideas to close off this issue. Thank you @edmondop

    I noticed that the original work from @andygrove referred examples in the docs and these seems to be pretty "unique" in the sense that no other doc page seems to take this approach , see https://github.com/apache/datafusion/tree/main/docs/src

    What do you think about moving those into examples and linking them from the docs? Do we have a preprocessor macro to includes the Rust code in a markdown upon compilation, or maybe to ensure that the embedded snippets compile and execute correctly ?

  6. alamb commented on Jun 3, 2024

    @alamb
    ContributorAuthor

    I think @andygrove started on a pre-processor that extracted the markdown and tested it here #7956

    Maybe we can revive that idea

  7. edmondop commented on Jun 3, 2024

    @edmondop
    Contributor

    I have created #10768 which I think would be a replacement of the Python scripts with something maintained somewhere else

  8. jc4x4 commented on Aug 16, 2024

    @jc4x4
    Contributor

    I've added a first draft to the doc. Let me know if you would look for a more sophisticated example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions