Skip to content

Implement memory management / limiting in SymmetricHashJoinExec #5636

Closed

Description

@alamb
          Actually there is also `SymmetricHashJoinExec` -- but, I suppose, its memory management is a bit more complicated and may be behind the scope of this issue:
  • It's, sort of, isolated -- at this moment there are no options for planning this operator with DataFusion planner
  • I don't think that simply throwing error and aborting execution is acceptable for SymmetricHashJoinExec -- if my understanding is correct -- main use case for this operator is joining two unbounded sources (streaming jobs), and from this point of view it doesn't make much sense to limit memory without any spilling fallbacks (subjectively, it doesn't seem correct to fail data streaming job in case of memory overallocation attempt)

My proposal here would be to file separate issue for SymmetricHashJoinExec memory management, and (as I see it) implement memory limitation along with data spilling. Maybe, we can go for it when we have reliable spilling for HashJoinExec, however, prior to it is also an option.

@ozankabak, @metesynnada it would be great to hear your thoughts on it.

Originally posted by @korowa in #5220 (comment)

Activity

  1. metesynnada commented on Mar 22, 2023

    @metesynnada
    Contributor

    I will be looking into this in a week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions