STO-67: fast-fail Model Repo storage quota check - #338
Conversation
45f8fcf to
36a185e
Compare
|
@cursor review |
|
Validated end-to-end against the real deployed dev environment (
All 12 relevant unit tests ( Note: this branch is currently 3 commits behind |
Adds a client-side check that fails fast, before creating the model or a single upload session, when a local upload would exceed the account's Model Repo storage quota. - api.GetModelRepoStorageUsage: wraps the server's modelRepoStorageUsage query (runpod/RunPod#5929). - checkModelRepoStorageQuota: compares the total local upload size against availableBytes and returns a clear, actionable error when it would be exceeded. Fails OPEN on any query error or unparseable response (older server, transient network issue) -- a client-side pre-check must never itself become the reason an otherwise-valid upload fails. Skipped entirely when enforcement is disabled server-side. - Wired into runAddModel right after the existing file-name/file-size validation, before addModelToRepo is called for either the --model-path (directory) or single-file (--file-name/--file-size) upload flows. - Storage-usage query failures go through modelRepoHTTPError/ modelRepoGraphQLError like the rest of the model-repo API surface, so access-denied and other server errors carry a stable error code (the STO-357 contract) rather than a bare string. Rebased onto current main to resolve conflicts with #339 (STO-366, pre-signed URL batching), which landed after this branch was created and fully migrated the local upload flow from a single createModelRepoUpload call per file to createModelRepoUploadBatch/CompleteModelRepoUploadBatch. Kept main's batch-based upload plumbing as-is and layered this PR's own additions on top (the getModelRepoStorageUsage var, checkModelRepoStorageQuota, and its call site in runAddModel); dropped this branch's now-superseded completeModelRepoUpload (singular) var, which #339 replaced with completeModelRepoUploadAll and which nothing in the current upload flow calls anymore. Testing: go build/vet/test clean across the whole module (all packages, not just cmd/model and api); govulncheck reports 0 vulnerabilities affecting this code. - cmd/model/modelRepoQuota_test.go covers the quota check's allow/deny/boundary/fail-open behavior and modelUploadRequestedBytes' directory-vs-flag size resolution. - api/model_test.go covers GetModelRepoStorageUsage response parsing, owner-variable omission, and graphql error surfacing, and the model-repo access-denied table test gains an entry for it (alongside #339's own new CreateModelRepoUploadBatch entry, both now coexisting). - One end-to-end test confirms addModelToRepo/createModelRepoUpload are never called when the fast-fail check rejects the upload. Revalidated live against the real dev environment (api.runpod.dev) after rebasing: over-quota model add still fast-fails client-side with zero server-side model rows created, under-quota model add still succeeds end-to-end through the now-batch-based upload path, and usage accounting updates correctly afterward.
36a185e to
0b9dc54
Compare
|
Resolved the merge conflicts against current Root cause: #339 fully migrated the local upload flow from a single How I resolved it: kept #339's batch-based upload plumbing as-is (nothing about that flow changes for this PR — the quota check runs before any upload session is created, batched or not), and layered this PR's own additions on top: the Validated before pushing:
Branch is now based on current |
|
bugbot run |
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 0b9dc54. Configure here.
Promptless documentation updates
|
Resolves STO-67.
Server-side counterpart: runpod/RunPod#5929.
#339 (STO-366, pre-signed URL batching) is independent — this check runs before
session creation either way.
Problem
runpodctlalready knows the total model size before uploading, but goesstraight into session creation with no check against the account's storage
quota. An over-quota upload should fail immediately with a useful message,
not after creating a model and partial upload sessions server-side.
Changes
api.GetModelRepoStorageUsagewraps the newmodelRepoStorageUsagequery.checkModelRepoStorageQuotacompares the total upload size againstavailableBytesand returns an actionable error when it would be exceeded.It fails open: a query error or unparseable response warns and proceeds,
since a client-side pre-check must never itself break a valid upload. Skipped
when enforcement is off (
Enforced: false) or no quota is set(
AvailableBytes: nil).createModelRepoUpload's server-side gate is stillthe real limit.
runAddModelafter file-name/file-size validation and beforeaddModelToRepo, covering both--model-pathand single-file flows.Testing
go vet ./...,go build ./...,go test ./...,govulncheck ./...checkModelRepoStorageQuotaallow/deny/boundary/fail-open, andmodelUploadRequestedBytes' directory-vs-flag size resolution.GetModelRepoStorageUsage, including its entry in theSTO-357 typed-error table.
addModelToRepoandcreateModelRepoUploadarenever reached when the check rejects the upload.
Note
Low Risk
Client-only pre-check that fails open on API errors; upload enforcement still happens server-side, with broad test coverage and no auth or data-model changes.
Overview
Adds a client-side Model Repo storage quota pre-check on
runpodctl model addso over-quota uploads fail before any server-side model or upload session is created.The API layer gains
ModelRepoStorageUsageandGetModelRepoStorageUsage, which calls the newmodelRepoStorageUsageGraphQL query (optionalowner; byte fields as strings). InrunAddModel, upload flows compute total bytes viamodelUploadRequestedBytes(--model-pathsum or--file-size) and runcheckModelRepoStorageQuotaafter upload flag validation and beforeaddModelToRepo. When enforcement is on andavailableBytesis known, uploads larger than available return an actionable error; query/parse failures fail open (stderr warning, upload continues). Server-side limits remain authoritative.Tests cover the API client, STO-357 access-denied table entry, quota allow/deny/boundary/fail-open behavior, and an integration test that
addModelToRepo/createModelRepoUploadare not called when quota rejects the upload.Reviewed by Cursor Bugbot for commit 0b9dc54. Configure here.