Intermittent limit.slt failure #9450

Activity

1. alamb

   added

   bugSomething isn't workingSomething isn't working

   on Mar 4, 2024
2. alamb commented on Mar 4, 2024

   

   alamb

   on Mar 4, 2024

   ContributorAuthor

   More actions

   I think the issue is that the partitioning of the test is not deterministic PARTITION BY t1.column has all the same values.

   # generate BIGINT data from 1 to 1000 in multiple partitions
   statement ok
   CREATE TABLE t1000 (i BIGINT) AS
   WITH t AS (VALUES (0), (0), (0), (0), (0), (0), (0), (0), (0), (0))
   SELECT ROW_NUMBER() OVER (PARTITION BY t1.column1) FROM t t1, t t2, t t3;
   
   # verify that there are multiple partitions in the input (i.e. MemoryExec says
   # there are 4 partitions) so that this tests multi-partition limit.
   query TT
   EXPLAIN SELECT DISTINCT i FROM t1000;
   ----
   logical_plan
   Aggregate: groupBy=[[t1000.i]], aggr=[[]]
   --TableScan: t1000 projection=[i]
   physical_plan
   AggregateExec: mode=FinalPartitioned, gby=[i@0 as i], aggr=[]
   --CoalesceBatchesExec: target_batch_size=8192
   ----RepartitionExec: partitioning=Hash([i@0], 4), input_partitions=4
   ------AggregateExec: mode=Partial, gby=[i@0 as i], aggr=[]
   --------MemoryExec: partitions=4, partition_sizes=[1, 2, 1, 1]

   I think we can fix this by changing the test to use different values.
3. alamb commented on Mar 4, 2024

   

   alamb

   on Mar 4, 2024

   ContributorAuthor

   More actions

   So I think the problem is that the input is hash partitioned into 4 partitions but somehow one of the partitions gets two batches and which partition gets the two batches is non deterministic

   explain CREATE TABLE t1000 (i BIGINT) AS
   WITH t AS (VALUES (0), (0), (0), (0), (0), (0), (0), (0), (0), (0))
   SELECT ROW_NUMBER() OVER (PARTITION BY t1.column1) FROM t t1, t t2, t t3;
   ----
   logical_plan
   CreateMemoryTable: Bare { table: "t1000" }
   --Projection: CAST(ROW_NUMBER() PARTITION BY [t1.column1] ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING AS Int64) AS i
   ----WindowAggr: windowExpr=[[ROW_NUMBER() PARTITION BY [t1.column1] ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING]]
   ------CrossJoin:
   --------CrossJoin:
   ----------SubqueryAlias: t1
   ------------SubqueryAlias: t
   --------------Values: (Int64(0)), (Int64(0)), (Int64(0)), (Int64(0)), (Int64(0))...
   ----------SubqueryAlias: t2
   ------------SubqueryAlias: t
   --------------Projection: 
   ----------------Values: (Int64(0)), (Int64(0)), (Int64(0)), (Int64(0)), (Int64(0))...
   --------SubqueryAlias: t3
   ----------SubqueryAlias: t
   ------------Projection: 
   --------------Values: (Int64(0)), (Int64(0)), (Int64(0)), (Int64(0)), (Int64(0))...
   

   Another way to fix the issue might be to add a configuration option such as datafusion.explain.show_statistics that would control if the partition_sizes were output in explain plan.

   Something like

   set datafusion.explain.show_sizes = false;

   And then the MemoryExec output would be generated without partition_sizesL

   MemoryExec: partitions=4
   

   ❤️React with ❤️2Reacted by Mustafa Akur and Jay Zhan
4. alamb

   changed the title [-]Intermittent limit.slt fails sometimes[/-] [+]Intermittent limit.slt failure[/+] on Mar 4, 2024
5. alamb commented on Mar 4, 2024

   

   alamb

   on Mar 4, 2024

   ContributorAuthor

   More actions

   This test was recently changed in #9444
6. mustafasrepo commented on Mar 4, 2024

   

   mustafasrepo

   on Mar 4, 2024

   Contributor

   More actions

   The second option seems like the most robust solution. Since in the current setup of the test, each time hash function changes, we might need to update test.
7. jayzhan211

   closed this as completedin #9474on Mar 6, 2024