Skip to content

Preventing OOM from binary store via endless chunked PUT - #4637

Merged
JoshVanL merged 17 commits into
dapr:mainfrom
WhitWaldo:prevent-oom-binary-store
Oct 13, 2026
Merged

JoshVanL merged 17 commits into
dapr:mainfrom
WhitWaldo:prevent-oom-binary-store

Conversation

@WhitWaldo

Copy link
Copy Markdown
Contributor

Description

Addressing opportunity for OOM on the sidecar if an in-memory binary store is presented an endless chunked PUT.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: dapr/dapr#10367 (comment)

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

  • Code compiles correctly
  • Created/updated tests
  • Extended the documentation
    • Created the dapr/docs PR:

Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
@WhitWaldo WhitWaldo self-assigned this Oct 12, 2026
@WhitWaldo
WhitWaldo requested review from a team as code owners October 12, 2026 00:32
Copilot AI balanced review requested due to automatic review settings October 12, 2026 00:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The maximum int64 value bypasses enforcement through overflow, and component metadata documentation is incomplete.

2 open findings
What changed in this PR

Limits in-memory binary uploads to prevent unbounded chunked requests from exhausting sidecar memory.

Changes:

  • Adds configurable maxSize with a 4 MiB default.
  • Rejects oversized uploads and adds boundary tests.
File Description
binarystore/​in-memory/​in_memory.go Adds and enforces upload limits.
binarystore/​in-memory/​in_memory_test.go Tests metadata parsing and size enforcement.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Outdated
kitmd "github.com/dapr/kit/metadata"
)

const defaultMaxSize int64 = 4 * 1024 * 1024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cap is per file. Total memory is still unbounded, so the OOM in the PR title is narrowed, not closed.

A loop of PUT /binarystore/mystore/f-$i with 4 MiB bodies passes the per-file check each time and each file is kept in s.files. Sidecar memory grows by 4 MiB per request until the kernel kills it. Repeated Overwrite: true PUTs of the same name are bounded; distinct names are not.

A total-bytes cap on the store, checked under the write lock, is the general fix:

if s.totalBytes-int64(len(existing))+int64(len(data)) > s.maxTotalSize {
	return binarystore.ErrStoreFull
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than a cap per store, how do you feel about having different stores implement different caps? Google and ADLSv2 have a 5 TB file cap and S3 supports a 50 TB cap. It seems a pity to limit all providers to the lowest common denominator especially as they're subject to change.

Rather, I'd like to see a reasonable cap on the in-memory provider (e.g. 512 MB; it's ok to cap it - it's only for testing) and put a maxObjectSize that establishes that upper bound for each (and then document each provider's limit).

I've implemented this under 278301a but I wanted to check with you before just resolving this conversation.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, per provider caps make sense to me

Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
…emory provider

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Copilot AI balanced review requested due to automatic review settings October 13, 2026 05:13
@codecov

codecov Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.25749% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.06%. Comparing base (cc03682) to head (98c1e74).
⚠️ Report is 70 commits behind head on main.

Files with missing lines Patch % Lines
binarystore/azure/blobstorage/blobstorage.go 26.66% 11 Missing ⚠️
binarystore/azure/datalake/datalake.go 15.38% 11 Missing ⚠️
binarystore/oci/objectstorage/client.go 50.00% 9 Missing ⚠️
binarystore/gcp/bucket/bucket.go 41.66% 5 Missing and 2 partials ⚠️
binarystore/oci/objectstorage/objectstorage.go 58.33% 3 Missing and 2 partials ⚠️
binarystore/aws/s3/metadata.go 75.00% 1 Missing and 1 partial ⚠️
binarystore/aws/s3/s3.go 60.00% 2 Missing ⚠️
binarystore/in-memory/in_memory.go 98.43% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4637      +/-   ##
==========================================
+ Coverage   31.94%   40.06%   +8.12%     
==========================================
  Files         353      354       +1     
  Lines       47723    36707   -11016     
==========================================
- Hits        15243    14706     -537     
+ Misses      31277    20617   -10660     
- Partials     1203     1384     +181     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Maximum-int metadata currently causes a failing test and integer overflow, while the documented interface guarantee is not implemented by every provider.

2 open findings
1 resolved since last review

🧠 Review effort: Balanced

Comment thread binarystore/store.go Resolved
…ze using a string unit size value. This ensures that users with higher memory thresholds are able to effectively write larger files than they might otherwise be limited to with a lower (but default) value of 16Mi.

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Copilot AI balanced review requested due to automatic review settings October 13, 2026 06:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

In-flight memory remains unbounded, and the Data Lake limits conflict with the SDK’s upload model and chunk-size maximum.

2 open findings
2 resolved since last review

🧠 Review effort: Balanced

Comment thread binarystore/azure/datalake/datalake.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Copilot AI balanced review requested due to automatic review settings October 13, 2026 07:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Several newly introduced provider-specific part-size parsing and validation paths lack direct test coverage.

0 open findings

2 resolved since last review
Previously missed (4)

In code that hasn't changed since last review

Low severity Test Azure Blob partSize parsing, propagation, and boundaries

binarystore/​azure/​blobstorage/​blobstorage.go:104

No Azure Blob test reaches the new partSize decoding or validates that the accepted value is propagated into the client and size limit; the existing metadata tests stop at missing required credentials. Please cover the default, a valid quantity, and both range boundaries, factoring this parsing into a helper if needed to avoid constructing a live client.

Low severity Test Data Lake partSize parsing, propagation, and boundaries

binarystore/​azure/​datalake/​datalake.go:110

No Data Lake test reaches the new partSize decoding or verifies propagation to UploadStreamOptions; the existing metadata tests only exercise missing account/filesystem fields. Please add coverage for the default, a valid quantity, and zero/over-4000-MiB values, factoring parsing into a helper if necessary.

Low severity Add metadata tests for partSize defaults, quantities, and validation

binarystore/​gcp/​bucket/​bucket.go:123

The new partSize default, quantity decoding, and 256 KiB validation are not exercised by TestParseMetadata; it still covers only bucket/auth fields. Please add cases for the default, a valid custom quantity, zero, and a non-multiple so changes cannot silently pass an invalid Writer.ChunkSize.

Low severity Test OCI partSize defaults, quantities, and size boundaries

binarystore/​oci/​objectstorage/​objectstorage.go:129

The OCI metadata tests do not cover this new partSize path. Please test the default and a valid byte-size quantity, plus values below 10 MiB and above 50 GiB, because these bounds determine both SDK buffering and the computed object-size limit.

🧠 Review effort: Balanced

Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Copilot AI balanced review requested due to automatic review settings October 13, 2026 07:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

OCI’s uploader does not propagate the limiting reader error, and Azure validates tuning after potentially creating remote resources.

1 open finding
Previously missed (2)

In code that hasn't changed since last review

Medium severity Validate partSize before creating the storage container

binarystore/​azure/​blobstorage/​blobstorage.go:99

partSize is validated only after CreateContainerStorageClient; that helper calls EnsureContainer by default. An invalid partSize can therefore create the remote container and perform network work even though Init ultimately fails. Decode and validate the upload tuning before creating the Azure client.

Medium severity Validate partSize before creating the filesystem

binarystore/​azure/​datalake/​datalake.go:105

partSize is validated only after CreateFileSystemStorageClient; that helper calls EnsureFileSystem by default. An invalid partSize can therefore create the remote filesystem and perform network work even though Init ultimately fails. Decode and validate the upload tuning before creating the Azure client.

🧠 Review effort: Balanced

Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/gcp/bucket/bucket.go Resolved
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Copilot AI balanced review requested due to automatic review settings October 13, 2026 15:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

In-memory overwrite and shutdown accounting can exceed or corrupt the configured capacity ceiling.

2 open findings
1 resolved since last review

🧠 Review effort: Balanced

Comment thread binarystore/in-memory/in_memory.go Resolved
Comment thread binarystore/in-memory/in_memory.go Resolved
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
Copilot AI balanced review requested due to automatic review settings October 13, 2026 16:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

OCI empty uploads regress and GCS part-size conversion is unsafe on supported 32-bit builds.

1 open finding
2 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Reject chunk sizes that overflow int

binarystore/​gcp/​bucket/​bucket.go:124

partSize is validated as int64 but later converted to int for storage.Writer.ChunkSize. This repository builds 32-bit GOARCH=arm, where an accepted value such as 4Gi wraps to zero; the GCS SDK interprets zero as disabling chunking and retries rather than using the requested size. Reject values that do not round-trip through the platform's int.

Low severity Document that ErrStoreFull may leave streams unread

binarystore/​store.go:51

The new contract omits the ErrStoreFull early-exit path: the in-memory store stops reading req.Data as soon as total capacity cannot be reserved. Document this exception too so callers do not assume the request stream was drained after ErrStoreFull.

🧠 Review effort: Balanced

NumberOfGoroutines: common.Int(binarystore.DefaultUploadConcurrency),
},
StreamReader: streamReader,
StreamReader: src,
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants