Repository navigation
pprof: handle allocation failures and backtrace re-entrancy - #1432
Merged
Merged
Conversation
The pprof profiler did not check its own allocations. A failed reallocation in the protobuf writer cleared the buffer pointer while the size kept growing, and module, string and address-map growth wrote through unchecked results, so an out-of-memory condition could corrupt or crash the profiled program. The pprof test also crashed on Linux: a sampled backtrace can allocate internally (on Linux `backtrace` calls `malloc`), and with mimalloc overriding `malloc` that allocation is sampled again, recursively capturing backtraces until the stack overflows. Make the snapshot best effort: allocation failures mark the protobuf buffer as failed, stop all further writes, and abort the snapshot without writing a (partial) file; module, string and address-map growth failures abort it the same way. Free the profiler heap when the profiler object cannot be allocated, destroy it (which frees the live profiler object, location table and base file name, unlike `mi_heap_delete` which would move them to the main heap and lose the only reference) when the location table cannot be initialized, free an unpublished location when its frame array cannot be allocated, and avoid the undefined 64-bit shift in `mi_exp_sample` when the random draw is 1. Guard against the re-entrancy with the profiler's spare per-theap field: a nested sample from our own stack capture is attributed to no location, but its size is still added to the global in-use total so that the matching `mi_free` stays balanced. The interval threshold is deliberately not advanced there, so a snapshot is never attempted from inside a backtrace capture. Register the pprof test on Linux as well, and add a deterministic out-of-memory test (Linux only, the platform where the arena exhaustion is verified) that exhausts the mimalloc arenas with OS allocation disabled and drains the profiler heap, so the next profiler allocation must fail: the snapshot must return normally and write no file, and snapshots must work again once memory is available. Validation: built with `-DMI_OVERRIDE=ON -DMI_PROFILE=ON -DMI_BUILD_TESTS=ON` on Linux and `ctest -R pprof` passes.
Daan (daanx)
added a commit
that referenced
this pull request
Oct 20, 2026
Collaborator
|
Thank you; I fixed the recursive invocation differently -- let me know how it works for you. |
Contributor
Author
|
Thanks, I tried it on Ubuntu 24.04 with profiling enabled, and test-pprof passed here. It works for me. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pprof profiler never checked its own allocations. A failed reallocation in the protobuf writer cleared the buffer pointer while the size kept growing, and module, string and address-map growth wrote through unchecked results, so running out of memory could corrupt or crash the profiled program.
There is a second problem on Linux: a sampled backtrace can allocate internally (
backtracecan callmalloc), and with mimalloc overridingmallocthat allocation is sampled again, recursively capturing backtraces until the stack overflows. The pprof test crashed this way on Linux, which went unnoticed because CMake only registered the test on macOS.The snapshot is now best effort. Allocation failures mark the protobuf buffer as failed, stop further writes and abort the snapshot without writing a partial file; the same applies to module, string and address-map growth. A failed profiler object allocation frees the private heap, and a failed location-table initialisation destroys it with
mi_heap_destroy, which frees the live blocks rather than moving them to the main heap and losing the reference. An unpublished location is freed when its frame array cannot be allocated, and the undefined 64-bit shift inmi_exp_sampleis avoided when the random draw is 1.Re-entrancy is detected per thread with the profiler's spare
theapfield. A nested sample is attributed to no location, but its size is still added to the global in-use total so the matching free stays balanced; the interval threshold is not advanced there, so a snapshot is never attempted from inside a backtrace capture.The pprof test is now registered on Linux, and a deterministic out-of-memory test exhausts the mimalloc arenas with OS allocation disabled and drains the profiler heap: the snapshot must return normally and write no file, and snapshots must work again once memory is available.
Built on Linux with
-DMI_OVERRIDE=ON -DMI_PROFILE=ON -DMI_BUILD_TESTS=ON;ctest -R pprofpasses. macOS and Windows have not been run locally.