Skip to content

Implement special Groups for StringViews #12771

Closed

Description

@alamb

Is your feature request related to a problem or challenge?

Part of #12680

In #12269 @jayzhan211 made significant improvements to how group values are stored in multi-column aggregations. This requires specialized implementations for different column types

His initial PR has implementations for PrimitiveArray and String/Binary. However it does not have a specialization for StringView

So that means that queries that group on multiple columns are even faster. This shows up by effectively slowing down some clickbench queries when they are run with StringView:

For example, this query is 10% slower with StringView

SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM 'hits.parquet' WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;

Describe the solution you'd like

I would like to make this (and similar) query faster when string view is enabled :

SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM 'hits.parquet' WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;

Note this is grouping by 2 columns

Here is how to reproduce the issue

Step 1. Get hits.parquet using bench.sh:

cd benchmarks
./bench.sh data clickbench_1

Step 2: Prepare a script with reproducer query:

set datafusion.execution.parquet.schema_force_view_types = true;

SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM 'hits.parquet' WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;

Step 3: Run query

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ datafusion-cli -f q.sql
  • set datafusion.execution.parquet.schema_force_view_types = true; --> Elapsed 0.688 seconds.
  • set datafusion.execution.parquet.schema_force_view_types = false; --> Elapsed 0.565 seconds.

Describe alternatives you've considered

I suggest implementing something like ByteViewGroupValueBuilder following the model of ByteGroupValueBuilder

The in progress values would be u128s and some buffers (maybe 2MB?)

implementing equal_to can take advantage of the inlined prefix optimization (aka compare the prefix inlined in the u128 and only check the value in the buffer if that is already equal)

Additional context

No response

Activity

  1. alamb commented on Oct 5, 2024

    @alamb
    ContributorAuthor

    BTW here are the flamegraphs:

    flamegraph-string
    flamegraph-stringview

    Screenshot 2024-10-05 at 9 26 48 AM

  2. changed the title [-]Implement special Groups or StringViews[/-] [+]Implement special Groups for StringViews[/+] on Oct 5, 2024
  3. alamb commented on Oct 5, 2024

    @alamb
    ContributorAuthor

    FYI @Rachelint / @jayzhan211 -- this might be an interesting project

  4. Rachelint commented on Oct 5, 2024

    @Rachelint
    Contributor

    Actually interesting, I am willing to help push it forward

  5. Rachelint commented on Oct 5, 2024

    @Rachelint
    Contributor

    take

  6. alamb commented on Oct 6, 2024

    @alamb
    ContributorAuthor

    I think it will be quite a cool optimization -- specifically checking for equal values can likely be optimized using the inlined prefix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions