Improve vectorized operations of GroupColumn #13275

Activity

1. Rachelint

   added

   enhancementNew feature or requestNew feature or request

   on Nov 6, 2024
2. Rachelint commented on Nov 28, 2024

   

   Rachelint

   on Nov 28, 2024

   ContributorAuthor

   More actions

   I tried the possible optimizations forvectorized_equal_to:
   #12996 (comment)
   #12996 (comment)

   But unfortunately seems no obvious improvement in clickbench, tpch and so on... although some improvement can see in benchmark for the function in #13539

   For take + eq, use eq in arrow kernel indeed can get performance improve, but the cost of take seems unacceptable...

   For optimizations for special case of input array (like all rows in the array needed, no nulls although it is nullable), it can see some improvement in the benchmark for function, but see no improvement in e2e benchmark(clickbench, tpch). I guess they may be not the hot path...

   I plan to not make further attempts recently...

   cc @Dandandan @alamb @jayzhan211
3. Rachelint commented on Nov 28, 2024

   

   Rachelint

   on Nov 28, 2024

   Last edited by Rachelint

   ContributorAuthor

   More actions

   I tried the possible optimizations forvectorized_equal_to: #12996 (comment) #12996 (comment)

   But unfortunately seems no obvious improvement in clickbench, tpch and so on... although some improvement can see in benchmark for the function in #13539

   For take + eq, use eq in arrow kernel indeed can get performance improve, but the cost of take seems unacceptable...

   For optimizations for special case of input array (like all rows in the array needed, no nulls although it is nullable), it can see some improvement in the benchmark for function, but see no improvement in e2e benchmark(clickbench, tpch). I guess they may be not the hot path...

   I plan to not make further attempts recently...

   cc @Dandandan @alamb @jayzhan211

   @LeslieKid maybe you can try it if some other possible optimizations not included above.
4. Dandandan commented on Nov 29, 2024

   

   Dandandan

   on Nov 29, 2024

   Last edited by Dandandan

   Contributor

   More actions

   Thanks @Rachelint for summarizing, that's interesting. One big difference between take + eq in join versus grouped aggregates seems to be that the ValueBuilders already are type erased, so maybe there is nothing to gain (and using take and eq just creates an extra copy.

   It seems for the join code, this could actually be used more (and joining on string columns might be not so fast at the moment).
5. Rachelint commented on Nov 29, 2024

   

   Rachelint

   on Nov 29, 2024

   ContributorAuthor

   More actions

   Thanks @Rachelint for summarizing, that's interesting. One big difference between take + eq in join versus grouped aggregates seems to be that the ValueBuilders already are type erased, so maybe there is nothing to gain (and using take and eq just creates an extra copy.

   It seems for the join code, this could actually be used more (and joining on string columns might be not so fast at the moment).

   🤔 I think maybe we can indeed try some solutions without take in join? @LeslieKid seems trying it.
6. Dandandan commented on Nov 29, 2024

   

   Dandandan

   on Nov 29, 2024

   Contributor

   More actions

   Yeah som experiments around that would be nice.

   Also see this issue:
   #12131

   👀React with 👀1Reacted by Yifan Su
7. Dandandan commented on Nov 29, 2024

   

   Dandandan

   on Nov 29, 2024

   Contributor

   More actions

   Oh I saw @LeslieKid already commented on that issue 👍

   🚀React with 🚀1Reacted by kamille
8. LeslieKid commented on Nov 29, 2024

   

   LeslieKid

   on Nov 29, 2024

   Contributor

   More actions

   🤔 I think maybe we can indeed try some solutions without take in join? @LeslieKid seems trying it.

   Yes, I am working on comparing arrays with indices in a for-loop (similar to the implementation of vectorized_equal_to() in multi group-by) instead of using take in hash join. See #13607