Skip to content

ANSI integral SUM overflow reports "integer overflow" without Spark's try_add suggestion #6661

Closed

Description

@andygrove

Describe the bug

Under spark.sql.ansi.enabled=true, an integral SUM overflow in Comet raises ARITHMETIC_OVERFLOW with the message parameter integer overflow and an empty alternative. Spark raises long overflow with the suggestion Use 'try_add' to tolerate overflow and return NULL instead.

Spark's SUM over integral input always returns LONG, and Sum adds through Add(left, right, evalContext) in both its update and merge expressions. So the overflow goes through MathUtils.addExact(Long, Long, context) with the try_add hint, the same on 3.4.3 through 4.2.0. Comet's native/spark-expr/src/agg_funcs/sum_int.rs raises arithmetic_overflow_error("integer") at its three overflow sites: the ungrouped update (which the ungrouped merge reuses), the grouped update, and the grouped merge.

Comet's decision to throw matches Spark. Only the message parameters differ, which matters to code that inspects getMessageParameters() or the message text.

This was part of item 2 in #5071, which also named SumInteger. #6217 assumed the rest of #5071 had been fixed, and #6249 fixes the binary +, - and * kernels but not SUM. Once #6249 removes the #6217 bullet, the compatibility guide no longer lists this divergence.

Steps to reproduce

SET spark.sql.ansi.enabled=true;
CREATE TABLE t (l BIGINT) USING parquet;
INSERT INTO t SELECT CASE WHEN id = 0 THEN 9223372036854775807 ELSE 1 END FROM range(0, 3, 1, 1);
SELECT SUM(l) FROM t;
-- Spark: [ARITHMETIC_OVERFLOW] long overflow. Use 'try_add' to tolerate overflow and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
-- Comet: [ARITHMETIC_OVERFLOW] integer overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

The final merge diverges the same way when each of two partitions holds part of the overflowing sum.

Expected behavior

The message parameter is long overflow and alternative carries the try_add suggestion, as in Spark.

Additional context

Reproduced locally on Spark 3.4 and 4.1 for both the partial and the merge path. With the function_name field that #6249 adds to SparkError::ArithmeticOverflow, raising ArithmeticOverflow { from_type: "long", function_name: "try_add" } at the three call sites made Comet match Spark's error class, parameters and message on both paths.

The existing ANSI support - SUM function test in CometAggregateSuite only checks that both messages contain ARITHMETIC_OVERFLOW, so it passes either way. It should compare the message parameters with Spark's.

Activity

  1. added
    bugSomething isn't working
    area:aggregationHash aggregates, aggregate expressions
    on Oct 5, 2026
  2. added
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed on Oct 5, 2026
  3. 0lai0 commented on Oct 6, 2026

    @0lai0
    Contributor

    take

  4. added a commit that references this issue on Oct 16, 2026
    53d398b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:aggregationHash aggregates, aggregate expressionsbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions