You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Preventing OOM from binary store via endless chunked PUT - #4637
Please make sure you've completed the relevant tasks for this PR, out of the following list:
Code compiles correctly
Created/updated tests
Extended the documentation
Created the dapr/docs PR:
Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.
The reason will be displayed to describe this comment to others. Learn more.
The cap is per file. Total memory is still unbounded, so the OOM in the PR title is narrowed, not closed.
A loop of PUT /binarystore/mystore/f-$i with 4 MiB bodies passes the per-file check each time and each file is kept in s.files. Sidecar memory grows by 4 MiB per request until the kernel kills it. Repeated Overwrite: true PUTs of the same name are bounded; distinct names are not.
A total-bytes cap on the store, checked under the write lock, is the general fix:
The reason will be displayed to describe this comment to others. Learn more.
Rather than a cap per store, how do you feel about having different stores implement different caps? Google and ADLSv2 have a 5 TB file cap and S3 supports a 50 TB cap. It seems a pity to limit all providers to the lowest common denominator especially as they're subject to change.
Rather, I'd like to see a reasonable cap on the in-memory provider (e.g. 512 MB; it's ok to cap it - it's only for testing) and put a maxObjectSize that establishes that upper bound for each (and then document each provider's limit).
I've implemented this under 278301a but I wanted to check with you before just resolving this conversation.
❌ Patch coverage is 71.25749% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.06%. Comparing base (cc03682) to head (98c1e74). ⚠️ Report is 70 commits behind head on main.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Maximum-int metadata currently causes a failing test and integer overflow, while the documented interface guarantee is not implemented by every provider.
…ze using a string unit size value. This ensures that users with higher memory thresholds are able to effectively write larger files than they might otherwise be limited to with a lower (but default) value of 16Mi.
Signed-off-by: Whit Waldo <whit.waldo@innovian.net>
No Azure Blob test reaches the new partSize decoding or validates that the accepted value is propagated into the client and size limit; the existing metadata tests stop at missing required credentials. Please cover the default, a valid quantity, and both range boundaries, factoring this parsing into a helper if needed to avoid constructing a live client.
Test Data Lake partSize parsing, propagation, and boundaries
binarystore/azure/datalake/datalake.go:110
No Data Lake test reaches the new partSize decoding or verifies propagation to UploadStreamOptions; the existing metadata tests only exercise missing account/filesystem fields. Please add coverage for the default, a valid quantity, and zero/over-4000-MiB values, factoring parsing into a helper if necessary.
Add metadata tests for partSize defaults, quantities, and validation
binarystore/gcp/bucket/bucket.go:123
The new partSize default, quantity decoding, and 256 KiB validation are not exercised by TestParseMetadata; it still covers only bucket/auth fields. Please add cases for the default, a valid custom quantity, zero, and a non-multiple so changes cannot silently pass an invalid Writer.ChunkSize.
Test OCI partSize defaults, quantities, and size boundaries
The OCI metadata tests do not cover this new partSize path. Please test the default and a valid byte-size quantity, plus values below 10 MiB and above 50 GiB, because these bounds determine both SDK buffering and the computed object-size limit.
partSize is validated only after CreateContainerStorageClient; that helper calls EnsureContainer by default. An invalid partSize can therefore create the remote container and perform network work even though Init ultimately fails. Decode and validate the upload tuning before creating the Azure client.
Validate partSize before creating the filesystem
binarystore/azure/datalake/datalake.go:105
partSize is validated only after CreateFileSystemStorageClient; that helper calls EnsureFileSystem by default. An invalid partSize can therefore create the remote filesystem and perform network work even though Init ultimately fails. Decode and validate the upload tuning before creating the Azure client.
partSize is validated as int64 but later converted to int for storage.Writer.ChunkSize. This repository builds 32-bit GOARCH=arm, where an accepted value such as 4Gi wraps to zero; the GCS SDK interprets zero as disabling chunking and retries rather than using the requested size. Reject values that do not round-trip through the platform's int.
Document that ErrStoreFull may leave streams unread
binarystore/store.go:51
The new contract omits the ErrStoreFull early-exit path: the in-memory store stops reading req.Data as soon as total capacity cannot be reserved. Document this exception too so callers do not assume the request stream was drained after ErrStoreFull.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Addressing opportunity for OOM on the sidecar if an in-memory binary store is presented an endless chunked PUT.
Issue reference
We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.
Please reference the issue this PR will close: dapr/dapr#10367 (comment)
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:
Note: We expect contributors to open a corresponding documentation PR in the dapr/docs repository. As the implementer, you are the best person to document your work! Implementation PRs will not be merged until the documentation PR is opened and ready for review.