Skip to content

RFC: Ideas for what to include in a user tutorial #842

Open

Description

@timsaucer

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In addition to the information we already have in the online documentation, it would be helpful to write a tutorial guiding a user through the various portions of DataFusion and how to get started. This issue is to collect ideas for what people would like to see in such a tutorial

Describe the solution you'd like

Please comment with topics that should be covered.

Additional context

Things I would like to see (unsorted list)

  • Creating a small dataframe from a pyarrow array
  • Reading/Writing data from/to csv and parquet
  • Zero copy import of data from pyarrow
  • Transferring DataFrame to/from pandas/polars
  • Displaying data via show(), repr_html(), and great tables
  • Basic column selection, including indexing into fields and element for structs and arrays
  • Performing joins
  • Performing window and aggregate functions, including how default and custom window frames work
  • Integrating with deltalake
  • Using object store from S3, Google Cloud, Azure
  • Unnesting columns
  • Making structs and arrays
  • Chaining DataFrame operations with transform (PR in review)
  • Doing a variety of conditional operations (both case and when without base statement)
  • Examples of string manipulation
  • Doing date time conversion
  • Writing a UDF (advanced topic: writing a rust UDF and using with datafusion-python)

Please add to the list what you would like to see!

Activity

  1. changed the title [-]RFC: Operations to include in a user tutorial[/-] [+]RFC: Ideas for what to include in a user tutorial[/+] on Aug 27, 2024
  2. mesejo commented on Aug 30, 2024

    @mesejo
    Contributor

    Great ideas!

    I'm curious about:

    Writing a UDF (advanced topic: writing a rust UDF and using with datafusion-python)

    Would this involve wrapping the rust UDF with PyO3?

    Reading/Writing data from/to csv and parquet

    I would also like to see how to read csv from HTTP directly

    Some other ideas:

    • Integration with numpy
    • UDF with numba acceleration (if possible?)
  3. timsaucer commented on Aug 30, 2024

    @timsaucer
    MemberAuthor

    Thanks! To answer your immediate question, I have a draft blog post about how you use rust UDFs here: https://github.com/timsaucer/datafusion-site/blob/tsaucer/python-udf-approaches/_posts/2024-08-06-datafusion-python-udf-comparisons.md I plan on publishing that soon-ish. I want to get a few things done with DF41 before we release it.

  4. mesejo commented on Aug 30, 2024

    @mesejo
    Contributor

    Thanks @timsaucer!

  5. timsaucer commented on Nov 22, 2024

    @timsaucer
    MemberAuthor

    There are a lot of suggestions in this twitter post from @ChristianCasazza
    https://x.com/casazzany/status/1860002216389550489?s=46

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions