Skip to content

Round Robin Polling Between Multiple Winners of Loser Tree in SPM #12231

Closed

Description

@berkaysynnada

Is your feature request related to a problem or challenge?

Assume a scenario where SPM (SortPreservingMergeExec) polls from 3 partitions, each of them gives the same value for the order expression for a very long time. Currently, we poll from the same partition until a new value comes from the other partitions. Continuously polling the same partition gets all upstream operator buffers and channels grow. However, we can execute a round-robin fashioned polling mechanism for the tied partitions.

Describe the solution you'd like

Loser tree can re-order the tied values of partitions.

Describe alternatives you've considered

No response

Additional context

This is the PR resolving a similar problem in the initialization phase: #12230

Activity

  1. athultr1997 commented on Oct 26, 2024

    @athultr1997
    Contributor

    Continuously polling the same partition gets all upstream operator buffers and channels grow

    @berkaysynnada Any idea why this happens ?

  2. berkaysynnada commented on Oct 27, 2024

    @berkaysynnada
    ContributorAuthor

    @berkaysynnada Any idea why this happens ?

    Because of eagerly polling mechanisms like RepartitionExec channels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions