Skip to content

model: fix iobuf over-reservation for null key/value records - #30980

Merged
WillemKauf merged 1 commit into
redpanda-data:devfrom
WillemKauf:record-builder-null-reserve-fix
Jul 1, 2026
Merged

model: fix iobuf over-reservation for null key/value records#30980
WillemKauf merged 1 commit into
redpanda-data:devfrom
WillemKauf:record-builder-null-reserve-fix

Conversation

@WillemKauf

@WillemKauf WillemKauf commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

append_record_to_buffer() reserved key_size() + value_size() bytes before writing a record. For a record whose key and value are both absent those sizes are -1 each, so the sum -2 underflows the size_t argument of iobuf::reserve_memory() to ~2^64. reserve_memory() then allocates 128 KiB fragments for every record (which is eventually trimmed back down to size).

This is especially problematic for cloud topics, since we call:

| `encode_placeholder_batch()`
 \-> `builder::add_raw_kv(std::nullopt, std::nullopt)` (for `N` records)
  \-> `builder::add_raw_kw()`
   \-> `model::append_record_to_buffer()`

This seems to mean potentially many (tens? hundreds?) of megabytes allocated and freed on the L0 write path for no reason.

Fix the bug by clamping reserved bytes to 0.

From a test, not checked in:

https://gist.github.com/WillemKauf/1090815d4e566165a0fe1e548ed877f2

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v26.1.x
  • v25.3.x
  • v25.2.x

Release Notes

Improvements

  • Fixes a bug where unnecessary memory allocations/deallocations were being made on the L0 write path in cloud topics

`append_record_to_buffer()` reserved `key_size() + value_size()` bytes
before writing a record. For a record whose key and value are both
absent those sizes are `-1` each, so the sum `-2` underflows the size_t
argument of `iobuf::reserve_memory()` to ~2^64. reserve_memory() then
trims the current fragment and allocates a fresh max-size (128 KiB)
fragment for every such record.

This is especially problematic for cloud topics, since we call:

| `encode_placeholder_batch()`
 \-> `builder::add_raw_kv()` (for `N` records)
  \-> `builder::add_raw_kw()`
   \-> `model::append_record_to_buffer()`

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a performance/pathological-allocation issue in model::append_record_to_buffer() where null key/value sizes (-1) could underflow when summed and be passed to iobuf::reserve_memory(), causing large fragment allocations on hot paths (notably cloud topics placeholder encoding).

Changes:

  • Clamp key/value reservation sizes to non-negative values before calling iobuf::reserve_memory().
  • Avoid size_t underflow on tombstone / null key+value records during record encoding.

Comment thread src/v/model/record_utils.cc
@WillemKauf
WillemKauf enabled auto-merge July 1, 2026 04:10
@vbotbuildovich

Copy link
Copy Markdown
Collaborator

CI test results

test results on build#86566
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FLAKY(PASS) ShadowLinkTopicFailoverTests test_producer_ids_failover {"storage_mode": "tiered_cloud"} integration https://buildkite.com/redpanda/redpanda/builds/86566#019f1bd5-9229-46a9-bc07-9b71fa2b08e2 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0066, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkTopicFailoverTests&test_method=test_producer_ids_failover
FLAKY(PASS) IdempotentProducerRecoveryTest test_java_client_recovery_on_producer_eviction null integration https://buildkite.com/redpanda/redpanda/builds/86566#019f1bd0-b254-4476-9ee8-5efc86a42b11 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=IdempotentProducerRecoveryTest&test_method=test_java_client_recovery_on_producer_eviction
FAIL src/v/security/tests/acl_store_fuzz src/v/security/tests/acl_store_fuzz unit https://buildkite.com/redpanda/redpanda/builds/86566#019f1bc3-95a6-4596-8e1e-c2353a34f216 0/1

@WillemKauf
WillemKauf merged commit 00e672d into redpanda-data:dev Jul 1, 2026
20 checks passed
@vbotbuildovich

Copy link
Copy Markdown
Collaborator

/backport v26.1.x

append_vint_to_iobuf(a, r.offset_delta());

a.reserve_memory(r.key_size() + r.value_size());
auto key_bytes = std::max(r.key_size(), 0);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do these methods return -1 and not 0?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-1 is used to distinguish a key/value in a record that is empty (std::nullopt) versus having a key/value that has a length of 0 ("").

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Back to 1999

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants