Skip to content

Add a GroupsAccumulator for the native regr_* aggregates #6886

Open

Description

@andygrove

What is the problem the feature request solves?

Regr in native/spark-expr/src/agg_funcs/regr.rs only implements Accumulator, so grouped regr_slope, regr_intercept, regr_r2, regr_sxx, regr_syy and regr_sxy run through DataFusion's GroupsAccumulatorAdapter. The adapter keeps one boxed accumulator per group and calls it once for each group present in a batch. In DataFusion 55.2 it also loops over every group's state on every batch (invoke_per_accumulator), so it falls further behind as the number of groups grows.

#4254 measured that path for the same building blocks. With 1M rows in 1000 groups in CometAggregateExpressionBenchmark, covar_samp took 93 ms and corr 105 ms through the adapter, against 56 ms and 64 ms with the aggregate left in Spark (the "Comet (Scan)" case). With GroupsAccumulators they took 26 ms and 32 ms.

The regr_* aggregates fall back to Spark in 1.1.0 (#6451). #6076 removes that fallback, so until Regr gets a GroupsAccumulator, grouped regr_* queries are likely to be slower than in 1.1.0.

Describe the potential solution

Implement groups_accumulator_supported and create_groups_accumulator for Regr, composing the existing grouped accumulators the way CorrelationGroupsAccumulator does:

  • regr_sxx and regr_syy: VarianceGroupsAccumulator with the central-moment update, evaluating to m2.
  • regr_sxy: CovarianceGroupsAccumulator, evaluating to the co-moment.
  • regr_slope and regr_intercept: CovarianceGroupsAccumulator plus a VarianceGroupsAccumulator over x, which only sees rows where both inputs are non-null when filter_var_by_pair_nulls is set (SPARK-48719).
  • regr_r2: CovarianceGroupsAccumulator plus two VarianceGroupsAccumulators with with_pearson_update(), keeping the SPARK-55969 flag and the captured evaluation mode.

The partial buffer layouts need to stay as they are. Adding the six functions to the statistical list in CometAggregateExpressionBenchmark would show where they land against Spark before and after.

Additional context

This should land before 1.2.0 (#6550). Found while reviewing #6076.

Activity

  1. added this to the 1.2.0 milestone on Oct 12, 2026
  2. modified the milestones: 1.2.0, 1.3.0 on Oct 15, 2026
  3. rich7420 commented on Oct 18, 2026

    @rich7420
    Contributor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:aggregationHash aggregates, aggregate expressionsenhancementNew feature or requestperformancepriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions