Improve performance of min/max aggregates for Durations via GroupsAccumulator #15317

Activity

1. alamb

   added

   enhancementNew feature or requestNew feature or request

   performanceMake DataFusion fasterMake DataFusion faster

   on Mar 19, 2025
2. alamb commented on Mar 19, 2025

   

   alamb

   on Mar 19, 2025

   ContributorAuthor

   More actions

   I think this is a pretty good first issue as it is well described and self contained
3. alamb commented on Mar 19, 2025

   

   alamb

   on Mar 19, 2025

   ContributorAuthor

   More actions

   BTW you can make durations in SQL like

   > create or replace table foo as select unnest (generate_series(now(), now() + interval '1 year', interval '1 day')) as "ts";
   0 row(s) fetched.
   Elapsed 0.005 seconds.
   
   > select now() - ts, arrow_typeof(now() - ts) as arrow_type from foo;
   +------------------------------------------------+----------------------+
   | now() - foo.ts                                 | arrow_type           |
   +------------------------------------------------+----------------------+
   | 0 days 0 hours 0 mins 26.882508000 secs        | Duration(Nanosecond) |
   | 0 days -23 hours -59 mins -33.117492000 secs   | Duration(Nanosecond) |
   | -1 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -2 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -3 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -4 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -5 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -6 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -7 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -8 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -9 days -23 hours -59 mins -33.117492000 secs  | Duration(Nanosecond) |
   | -10 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -11 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -12 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -13 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -14 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -15 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -16 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -17 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -18 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -19 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -20 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -21 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -22 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -23 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -24 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -25 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -26 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -27 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -28 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -29 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -30 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -31 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -32 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -33 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -34 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -35 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -36 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -37 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | -38 days -23 hours -59 mins -33.117492000 secs | Duration(Nanosecond) |
   | .                                                                     |
   | .                                                                     |
   | .                                                                     |
   +------------------------------------------------+----------------------+
   366 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
   Elapsed 0.005 seconds.
4. alamb

   changed the title [-]Improve performance if min/max aggregates for Durations[/-] [+]Improve performance of min/max aggregates for Durations via GroupsAccumulator[/+] on Mar 19, 2025
5. shruti2522 commented on Mar 19, 2025

   

   shruti2522

   on Mar 19, 2025

   Contributor

   More actions

   take

   ❤️React with ❤️1Reacted by Andrew Lamb
6. github-actions

   assigned

   shruti2522

   on Mar 19, 2025
7. alamb commented on Mar 19, 2025

   

   alamb

   on Mar 19, 2025

   ContributorAuthor

   More actions

   Thanks @shruti2522 -- you are crushing it of late

   ❤️React with ❤️1Reacted by Shruti Sharma
8. alamb

   closed this as completedin #15322on Mar 21, 2025