Skip to content

Consolidate Example: simplify_udaf_expression.rs into advanced_udaf.rs #13842

Description

@alamb

Is your feature request related to a problem or challenge?

The large number of small examples in DataFusion leads to multiple problems:

  • Unnecessarily large amounts of temporary disk space (each example binary takes 100+ MB)
  • Larger cargo t time (each binary must be linked / created)
  • Makes it harder to find the relevant examples

Thus I think the fewer examples we have the

  1. faster our dev experience will be
  2. easier it will be to find the relevant examples

Describe the solution you'd like

Then update the readme page:

Describe alternatives you've considered

No response

Additional context

I think this is a good first issue as the task is straightforward and doesn't require deep understanding of the DataFusion codebase

Activity

  1. changed the title [-]Consolidate parquet.sql into [/-] [+]Consolidate Example: simplify_udaf_expression.rs into advanced_udaf.rs[/+] on Dec 19, 2024
  2. takaebato commented on Dec 20, 2024

    @takaebato
    Contributor

    take

  3. alamb commented on Dec 20, 2024

    @alamb
    ContributorAuthor

    Thank you @takaebato 🙏

  4. takaebato commented on Dec 24, 2024

    @takaebato
    Contributor

    @alamb

    Sorry for the delay🙏 But I have a quick question!
    I’m not too sure about this, but should the return type of AggregateFunctionSimplification perhaps be Result<ExprSimplifyResult>(or a similar enum that can indicate whether or not simplification occurred)?

    Currently, it doesn’t seem possible to skip simplification, as it always returns a simplified Expr (or Err).
    The simplify method can return an Option, but the decision to simplify seems to rely on information passed to AggregateFunctionSimplification, so might it be better if this method can choose whether or not to simplify..?

  5. alamb commented on Dec 24, 2024

    @alamb
    ContributorAuthor

    @alamb

    Sorry for the delay🙏 But I have a quick question! I’m not too sure about this, but should the return type of AggregateFunctionSimplification perhaps be Result<ExprSimplifyResult>(or a similar enum that can indicate whether or not simplification occurred)?

    Currently, it doesn’t seem possible to skip simplification, as it always returns a simplified Expr (or Err). The simplify method can return an Option, but the decision to simplify seems to rely on information passed to AggregateFunctionSimplification, so might it be better if this method can choose whether or not to simplify..?

    The rationale for returning ExprSimplifyResult is so that the argument doesn't have to be cloned -- aka to avoid coping the ownership of argument is passed to the function and there needs to be some way to get the original structuer back

  6. takaebato commented on Dec 26, 2024

    @takaebato
    Contributor

    @alamb

    Thank you so much for the explanation!!
    (I also realized that if we want to use the actual implementation conditionally, we can return Ok(Expr::AggregateFunction(aggregate_function)).)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions