CometHashAggregate prefixed with ! in explain plan #2214

Activity

1. rishvin

   added

   bugSomething isn't workingSomething isn't working

   on Aug 22, 2025
2. andygrove

   added this to the 0.11.0 milestone on Sep 8, 2025
3. andygrove

   assigned

   comphead

   on Sep 9, 2025
4. andygrove

   modified the milestones: 0.11.0, 0.12.0 on Sep 9, 2025
5. andygrove commented on Sep 15, 2025

   

   andygrove

   on Sep 15, 2025

   Last edited by andygrove

   Member

   More actions

   I started debugging this, and CometHashAggregateExec.missingInputs contains values, which leads to the warnings.

   In CometHashAggregateExec, I updated the code:

     override def stringArgs: Iterator[Any] =
       Iterator(input, mode, groupingExpressions, aggregateExpressions, child, "missing", missingInput)
   

   Here is an example output:

      !CometHashAggregate [_2#7, count#25L], Final, [_2#7], [count(_1#6)], missing, {count(_1)#21L, count(_1#6)#20L}
      +- CometExchange hashpartitioning(_2#7, 10), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=211]
         +- !CometHashAggregate [_1#6, _2#7], Partial, [_2#7], [partial_count(_1#6)], missing, {count#25L}
            +- CometScan [native_iceberg_compat] parquet [_1#6,_2#7] ... ReadSchema: struct<_1:int,_2:int>
   

   👍React with 👍2Reacted by Rishab Joshi and Oleks V
6. andygrove

   modified the milestones: 0.12.0, 0.11.0 on Sep 15, 2025
7. comphead commented on Sep 15, 2025

   

   comphead

   on Sep 15, 2025

   Contributor

   More actions

   I remember hitting this ! on other operators, like CometTakeOrderedAndProject, checking it
8. comphead commented on Sep 15, 2025

   

   comphead

   on Sep 15, 2025

   Contributor

   More actions

   @rishvin does it happen with AQE only?
9. comphead commented on Sep 15, 2025

   

   comphead

   on Sep 15, 2025

   Contributor

   More actions

   test("invalid CometHashAggregate") {
       sql("CREATE TABLE t(id int, value int) USING PARQUET")
       sql("INSERT INTO t VALUES (1, 10)")
       sql("INSERT INTO t VALUES (2, 20)")
       sql("INSERT INTO t VALUES (3, 10)")
       sql("INSERT INTO t VALUES (4, 30)")
       sql("INSERT INTO t VALUES (5, 20)")
       sql("INSERT INTO t VALUES (6, 10)")
   
       sql("select * from t").write.mode("overwrite").parquet("/tmp/test_11")
   
       withSQLConf(
         CometConf.COMET_EXEC_SHUFFLE_ENABLED.key -> "true",
         "spark.comet.cast.allowIncompatible" -> "true",
         "spark.comet.enabled" -> "true",
         "spark.comet.exec.replaceSortMergeJoin" -> "true",
         "spark.comet.exec.shuffle.enableFastEncoding" -> "true",
         "spark.comet.exec.shuffle.enabled" -> "true",
         "spark.comet.exec.shuffle.fallbackToColumnar" -> "true",
         "spark.comet.explain.native.enabled" -> "true",
         "spark.comet.explainFallback.enabled" -> "true",
         "spark.shuffle.manager" -> "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager",
         "spark.comet.logFallbackReasons.enabled" -> "true") {
         spark.read.parquet("/tmp/test_11").createOrReplaceTempView("t2")
         spark.sql("select count(id) from t2 group by id").explain("extended")
         sql("drop table t")
       }
   
   
   == Physical Plan ==
   AdaptiveSparkPlan isFinalPlan=false
   +- !CometHashAggregate [id#43, count#51L], Final, [id#43], [count(id#43)]
      +- CometExchange hashpartitioning(id#43, 10), ENSURE_REQUIREMENTS, CometNativeShuffle, [plan_id=141]
         +- !CometHashAggregate [id#43], Partial, [id#43], [partial_count(id#43)]
            +- CometScan [native_iceberg_compat] parquet [id#43] Batched: true, DataFilters: [], Format: CometParquet, Location: InMemoryFileIndex(1 paths)[file:/tmp/test_11], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>
   
   
   

   This is interesting as ! usually says the reference from preceding plan node is missing, which looks okay here

   and it still warns without AQE
10. comphead commented on Sep 15, 2025

    

    comphead

    on Sep 15, 2025

    Last edited by comphead

    Contributor

    More actions

    Spark considers count#51L to be as input, and thinks it is missing. This is the reason for exclamation mark

    UPD: Spark HashAggregate.producedAttributes contains count values, where CometHashAggregate does not, refer to the code below, that's causing the exclamation mark.

    
      final def missingInput: AttributeSet = references -- inputSet
    
      lazy val references: AttributeSet = {
        AttributeSet.fromAttributeSets(expressions.map(_.references)) -- producedAttributes
      }
    
    

    It does seem it is not an issue with wrong plan
    @andygrove should we address this in 0.11.0?

    👍React with 👍1Reacted by Rishab Joshi
11. andygrove commented on Sep 15, 2025

    

    andygrove

    on Sep 15, 2025

    Member

    More actions

    Spark considers count#51L to be as input, and thinks it is missing. This is the reason for exclamation mark

    UPD: Spark HashAggregate.producedAttributes contains count values, where CometHashAggregate does not, refer to the code below, that's causing the exclamation mark.

    
      final def missingInput: AttributeSet = references -- inputSet
    
      lazy val references: AttributeSet = {
        AttributeSet.fromAttributeSets(expressions.map(_.references)) -- producedAttributes
      }
    

    It does seem it is not an issue with wrong plan @andygrove should we address this in 0.11.0?

    Yes, I think it would be good to fix this for 0.11.0.

    👍React with 👍2Reacted by Oleks V and Rishab Joshi
12. mbutrovich

    closed this as completedin #2409on Sep 17, 2025