Skip to content

CometMemoryPool may release more memory than acquired memory #1733

Closed
#6128
#1732
Closed
CometMemoryPool may release more memory than acquired memory#1733
#6128
#1732
Labels
bugSomething isn't working

Description

@wForget
wForget
opened on May 12, 2025

Describe the bug

The memory acquired by CometMemoryPool.grow may be less than the actual request, so CometMemoryPool.shrink may release more memory than acquired memory.

datafusion-comet/native/core/src/execution/memory_pools/unified_pool.rs

Lines 93 to 95 in 25e39ab

self.acquire(additional)
.unwrap_or_else(|_| panic!("Failed to acquire {} bytes", additional));
self.used.fetch_add(additional, Relaxed);

We may have two improvement options:

  1. Check acquired memory when CometMemoryPool grows: #1732 (I prefer this one)
  2. Add an acquired counter to avoid releasing more unacquired memory : #1731

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

  1. wForget
    added
    bugSomething isn't working
    on May 12, 2025
  2. wForget
    changed the title [-]Check acquired memory when CometMemoryPool grows[/-] [+]CometMemoryPool may release more memory than acquired memory[/+] on May 12, 2025
  3. andygrove
    closed this as completedin #1732on May 12, 2025
  4. comphead
    added a commit that references this issue on Sep 23, 2026

    fix: let Comet memory pools overcommit on grow instead of panicking (a…

    d3406c6
  5. andygrove
    added a commit that references this issue on Sep 25, 2026

    fix: let Comet memory pools overcommit on grow instead of panicking (#…

    f037304
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions