Repository navigation
[miniflare] Manage local Artifacts repositories through the Local Explorer - #16968
dinasaur404 wants to merge 1 commit into
Conversation
…lorer API Start local Artifacts namespaces without Worker bindings in explorer and shared-storage mode, and serve the Artifacts REST routes from the Local Explorer so `cf --local` can create, list, fork, import and delete repositories and manage repository tokens.
🦋 Changeset detectedLatest commit: a99ef69 The changes in this PR will be included in the next version bump. This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Codeowners approval required for this PR:
Show detailed file reviewers
|
| for (const namespace of artifactsExplorerNamespaces.unbound) { | ||
| const local = await createLocalNamespaceRuntime( | ||
| root, | ||
| namespace, | ||
| artifactsController, | ||
| artifactsEventSubscriptions, | ||
| { bound: false } | ||
| ); |
There was a problem hiding this comment.
🔴 Concurrent repository creation loses metadata
When cf --local and wrangler dev create different repos concurrently, their exclusive queues cannot coordinate across processes. Both save the entire repos map; the later save can erase the earlier repo while leaving its Git data behind.
Learn more
Each running Miniflare process owns its own Artifacts Durable Object and its own in-memory mutation queue. Both processes now open the same metadata disk directly. exclusive serializes only one object's operations, and create reads and rewrites the complete repository map. A simultaneous create in another process can save a map built before the first create committed, removing its metadata while leaving its Git directory on disk. The same risk applies to simultaneous token changes, deletes, and push metadata updates.
Example: A dev server and cf --local both load {}. The server creates alpha, while cf --local creates beta. If cf --local saves last, the persisted map contains only beta, although alpha's Git directory still exists.
Recommended fix: Route Artifacts namespace mutations through one storage owner per persistence root, or add cross-process transaction/locking that reloads the map inside the lock before each mutation and commits atomically. Cover simultaneous creates and token changes from two Miniflare instances.
Was this helpful? React with 👍 or 👎 to provide feedback.
| for (const namespace of artifactsExplorerNamespaces.unbound) { | ||
| const local = await createLocalNamespaceRuntime( | ||
| root, | ||
| namespace, | ||
| artifactsController, | ||
| artifactsEventSubscriptions, | ||
| { bound: false } | ||
| ); | ||
| sockets.push(local.socket); |
There was a problem hiding this comment.
🔴 New simulator deletes active Git staging
Starting cf --local during a dev server import or fork starts another sidecar for the same Git root. Its startup cleanup removes the active staging directory, aborting the operation.
Learn more
An Artifacts sidecar operates on one Git storage directory. An import or fork builds a new repository in a staging directory before moving it into place. startGitSidecar unconditionally deletes staging directories at startup, even if another process is actively using them. Starting an unbound namespace creates another sidecar for the same root, so ordinary cf --local startup can erase a running dev server's staging directory. Creation locks do not guard the startup cleanup.
Example: wrangler dev starts importing demo into .artifacts-stage-ABC123; a user invokes cf artifacts namespaces repos list --local against the same state. The new sidecar deletes .artifacts-stage-ABC123, and the import fails.
Recommended fix: Coordinate sidecar startup and staging cleanup across processes; only remove staging paths proven abandoned. A shared per-root sidecar or cleanup that checks an active ownership lock can avoid deleting another process's work.
Was this helpful? React with 👍 or 👎 to provide feedback.
| validateQuery(zArtifactsReposListData.shape.query.unwrap()), | ||
| (c) => listArtifactsRepos(c, c.req.param("namespace"), c.req.valid("query")) |
There was a problem hiding this comment.
🟡 Invalid Artifacts queries return generic errors
When limit, sort, state, or page is invalid, validateQuery returns code 10001 without Artifacts error metadata. Artifacts clients receive a different error contract from the new body and operation routes.
Learn more
The generic validateQuery uses validationHook, which emits code 10001 without documentation_url or source.pointer. Both new repository and token list routes use it, whereas validateArtifactsBody emits Artifacts-specific errors. A request that fails generated query validation never reaches the Artifacts service or its error mapping.
Example: GET /artifacts/namespaces/default/repos?limit=0 returns a generic 400 with error 10001; an invalid repository create body returns an Artifacts 400 with its specific numeric code and documentation link.
Recommended fix: Add an Artifacts query validator that uses the same generated schema and coercion as validateQuery but converts errors to the Artifacts envelope, including the field pointer. Apply it to both repository and token list routes.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const updatedAt = repositories | ||
| .map((repository) => repository.updatedAt) | ||
| .reduce((latest, time) => (time > latest ? time : latest), createdAt); |
There was a problem hiding this comment.
🟡 Namespace update timestamp regresses after deletion
After deleting the most recently updated repo, explorerSummary computes updatedAt from only the remaining live repos. The namespace timestamp moves backward, so clients using it to detect changes miss the deletion.
Learn more
The namespace summary calculates its update time from repositories still present in the persisted map. finishDeletion removes a deleted repository from that map, and does not retain the deletion time. Removing the most recently updated repository therefore makes the namespace's reported time move backward. For an empty namespace it returns the namespace creation time even after later deletions.
Example: A namespace is created at 10:00, and its sole repo is created at 11:00. Deleting the repo at 12:00 changes updated_at from 11:00 back to 10:00.
Recommended fix: Persist a namespace-level modification timestamp with repository mutations, including deletions, and use it for namespace summaries rather than deriving it solely from currently live repositories.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
💬 Approved with comments — No blocking issues. One WARNING: the docs overstate which routes return 503 when Git is missing. There are also two INFO notes: the namespace list returns an empty 200 when Git is missing, and the namespace marker write is not atomic.
Findings: 1 warning · 2 info (3 are inline comments on the diff)
Not opened by the correctness review: 2 changed files it was meant to read
packages/miniflare/src/workers/local-explorer/generated/types.gen.tspackages/miniflare/src/workers/local-explorer/generated/zod.gen.ts
The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.
🔧 Fix with your agent
Paste the following into your coding agent to investigate and fix these issues.
Fix the following code review issues found in pull request #16968.
Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order
## WARNING (1)
### [Documentation] Docs say all Artifacts routes return 503 when Git is missing, but the namespace list returns 200
- **File:** packages/miniflare/src/plugins/artifacts/LOCAL_DEVELOPMENT.md:179
## INFO (2)
### [Code Quality] Namespace list returns an empty 200 instead of 503 when Git is missing
- **File:** packages/miniflare/src/workers/local-explorer/resources/artifacts.ts:303
### [Code Quality] `namespace.json` is written non-atomically, so a concurrent start can reset `createdAt`
- **File:** packages/miniflare/src/plugins/artifacts/storage.ts:64
📋 Code Quality: 💬 2 info
No bugs found. Two low-impact edge cases: the namespace list when Git is missing, and the non-atomic marker write.
Recommendation: 💬 Optional Nits
- [INFO] Namespace list returns an empty 200 instead of 503 when Git is missing (
packages/miniflare/src/workers/local-explorer/resources/artifacts.ts:303) - [INFO]
namespace.jsonis written non-atomically, so a concurrent start can resetcreatedAt(packages/miniflare/src/plugins/artifacts/storage.ts:64)
View full review
- When Git is missing,
GET /artifacts/namespacesreturns an empty 200 list, while the per-namespace routes return 503 (10401) throughgetNamespace. recordLocalNamespacewritesnamespace.jsonwith a plainwriteFile. If two processes start at once on the same state directory,createdAtcan be overwritten.
📝 Documentation: ⚠️ 1 warning
The docs overstate which routes return 503 when Git is unavailable.
Recommendation: 📝 Minor Comments
- [WARNING] Docs say all Artifacts routes return 503 when Git is missing, but the namespace list returns 200 (
packages/miniflare/src/plugins/artifacts/LOCAL_DEVELOPMENT.md:179)
View full review
LOCAL_DEVELOPMENT.md says a missing Git installation makes the Artifacts routes return 503. The namespace-list route returns a successful, possibly empty list instead.
✅ All other checks passed
🔒 Security: ✅ LGTM
Namespaces are looked up with Object.hasOwn against the known bindings, so a request cannot pick an arbitrary path. Namespace markers are checked against the hashed directory name, and request bodies are schema-validated.
📖 AGENTS.md: ✅ LGTM
No issues found.
⚡ Performance: ✅ LGTM
The new explorer paths do in-memory scans over small local datasets, plus a few local Git and filesystem calls.
Reviewed commit: a99ef691
|
|
||
| These commands share state with `wrangler dev --persist-to .wrangler/state`, in either direction, including while the dev server runs. `cf --local` can use the `default` namespace in an empty state directory, plus any namespace that already has local state. A namespace is recorded the first time a dev server with this version starts with a binding to it; state from an earlier version appears once you have done that. Local namespaces cannot be created or deleted through the API: create one by binding it in your Worker. | ||
|
|
||
| The local explorer of a running dev server (`/cdn-cgi/local/explorer/api`) serves the same routes for the namespaces it binds and those already in its state directory. It does not create `default` unless a Worker binds it. A missing Git installation disables only these routes, which return `503`, not the dev server. |
There was a problem hiding this comment.
Docs say all Artifacts routes return 503 when Git is missing, but the namespace list returns 200
This line says a missing Git installation disables these routes, which then return 503. GET /artifacts/namespaces still returns a 200 with the namespaces that are available, which may be an empty list. Only requests for an unavailable namespace return 503. Either narrow the wording or make the list route return 503 as well.
| } | ||
|
|
||
| /** Lists every local namespace; local listings are not paginated. */ | ||
| export async function listArtifactsNamespaces( |
There was a problem hiding this comment.
ℹ️ INFO · Code Quality
Namespace list returns an empty 200 instead of 503 when Git is missing
When Git is unavailable, the map has no unbound namespaces and unavailableReason is set. getNamespace turns that into a 503 (10401), but this handler ignores unavailableReason. So cf artifacts namespaces list --local returns [] with a 200, even though default is supposed to exist in shared-storage mode. Consider returning the same 503 here when unavailableReason is set and no namespaces are available, so users see why the list is empty.
| return existing; | ||
| } | ||
| const record = { namespace, createdAt: new Date().toISOString() }; | ||
| await writeFile(marker, `${JSON.stringify(record)}\n`); |
There was a problem hiding this comment.
ℹ️ INFO · Code Quality
namespace.json is written non-atomically, so a concurrent start can reset createdAt
A wrangler dev process and a cf --local process can start at the same time against the same state directory. One of them can then read a half-written marker. The parse error is swallowed, so that process rewrites the file with a new createdAt. The only effect is cosmetic (created_at is reported inconsistently), but writing to a temp file and renaming it, or using an exclusive-create flag for the first write, would avoid it.
@cloudflare/autoconfig
@cloudflare/build-output-utils
@cloudflare/codemods
@cloudflare/config
@cloudflare/containers-shared
create-cloudflare
@cloudflare/deploy-helpers
@cloudflare/kv-asset-handler
miniflare
@cloudflare/pages-functions
@cloudflare/pages-shared
@cloudflare/runtime-types
@cloudflare/unenv-preset
@cloudflare/vite-plugin
@cloudflare/vitest-plugin
@cloudflare/workers-auth
@cloudflare/workers-editor-shared
@cloudflare/workers-utils
wrangler
commit: |
There was a problem hiding this comment.
Thanks @dinasaur404! this closes a really useful gap for cf --local. The route set, the error envelope, the result shapes and the regenerated OpenAPI all look right. Reusing the binding's create/fork/import/token paths instead of writing a parallel version was a great call. What I'd like to sort out before this ships is how several processes share one namespace's storage, not the API itself.
What I saw: each process that starts a namespace opens its metadata and Git storage directly, so no single process is in charge of a namespace. When I tried it, I could confirm Devin's two concurrency notes. With wrangler dev running, I sent 10 creates through the dev explorer and 10 through a cf --local-style instance at the same time. That gave 19×201 and 1×500, but only 12 repos were listed afterwards. Seven repos that got a 201 went missing, and their Git directories stayed on disk. Ordinary dev sessions have the explorer on by default, so they now also start every namespace recorded in the state directory. That means this can happen even without cf. A second wrangler dev on the same --persist-to with no Artifacts binding started a Git backend for default and served its repos.
Suggested changes:
- One owner per namespace's storage (
plugins/artifacts/index.ts,createLocalNamespaceRuntime/getLocalArtifactsExplorerNamespaces; the README paragraph "not routed through a shared storage owner"). When a running session already serves a namespace, the explorer could forward the call to that session'sLocalArtifactsExplorerthrough the dev registry, asaggregation.tsalready does for peer entrypoints. It would open storage directly only when no one serves it. As a bonus,cf --localchanges would then reach the running session'striggers.events, and the returnedremotewould point at a live Git listener. A test that sends concurrent creates through both instances and checks that every 201 is listed would lock this in. - Keep ordinary dev sessions to the namespaces they bind (
getLocalArtifactsExplorerNamespaces). Unbound namespaces could start only in shared-storage mode, or lazily, so a session that doesn't use Artifacts never runs a Git backend on another project's state. The test "a dev session's explorer manages its bound namespace only" (artifacts.spec.ts:502) uses an empty state directory. Could it also cover a directory that already has a namespace? explorerDeleteand tombstones (workers/artifacts/binding.worker.ts). For a repo that is already marked for deletion, it returnsnull, so the route answers 404 and doesn't retry cleanup. The binding'sdelete()reports success and finishes the deletion. It also reads the repo's ID outsideexclusive(). Maybe#deletecould return the removed repository's ID, so both callers share one path?- Tests for the events behaviour in the docs (
LOCAL_DEVELOPMENT.md). Both "a dev server's explorer publishes to its subscriptions" and "cf --localpublishes nothing" would be good to pin down, especially since point 1 changes the second. - A few small ones:
explorerListrepeats#list's limit/cursor/filter logic. One list with sort/search options and two output shapes would keep them in step.listArtifactsRepos(resources/artifacts.ts) addsstatusto each item, but the production REST list schema doesn't have that field.- The
workerIndex === 0check in the plugin'sgetServicesstarts instance-wide namespaces from per-worker code. It might read more clearly next to the global services.
The changeset looks good as a minor. Once point 1 lands, the remote and events sentences in it and in LOCAL_DEVELOPMENT.md will want a small update.
Fixes
cf artifacts ... --local, which today fails with "This command has no local equivalent ... The local explorer API does not implement POST /artifacts/namespaces/default/repos." KV, D1, R2, Workflows and Flagship already work withcf --local.cfsends every--localrequest to Miniflare's Local Explorer. It starts Miniflare with no Worker bindings, onlyunsafeLocalExplorer,unsafeEnableSharedStorage,resourcePersistencePathandisolatedResourcePersistencePath. The Artifacts plugin only started namespaces listed inartifactsbindings, and the explorer had no Artifacts routes.Changes
Namespaces without bindings. In explorer or shared-storage mode, the Artifacts plugin also starts local namespaces with no binding. Namespace directories are hashed, so each namespace now records its name in
namespace.json, and Miniflare starts every recorded namespace. In shared-storage mode (cf --local),defaultis always included, so creating a repository works in an empty state directory. Plainwrangler dev/Vite sessions (explorer on by default) do not createdefault, so sessions that don't use Artifacts start no Git sidecar and write no Artifacts state.Missing Git disables only these unbound namespaces; their routes return
503(10401). It does not fail startup or the other explorer resources.Explorer entrypoint. A new
LocalArtifactsExplorerentrypoint per namespace reuses the simulator's create, fork, import, list, delete, createToken and revokeToken. It adds the REST-only views: repository IDs, namespace summaries, forkobjectscount (from the Git sidecar), and token revocation by ID across the namespace. Each namespace's service is registered inLOCAL_EXPLORER_BINDING_MAP.artifacts, following the Flagship pattern.Routes (public REST paths without
/accounts/{account_id}):GET /artifacts/namespaces,GET /artifacts/namespaces/:nsGET|POST /artifacts/namespaces/:ns/repos,GET|DELETE .../repos/:namePOST .../repos/:name/fork,POST .../repos/:name/importGET .../repos/:name/tokens,POST /artifacts/namespaces/:ns/tokens,DELETE /artifacts/namespaces/:ns/tokens/:idRequest and response shapes, statuses, numeric error codes,
source.pointer,documentation_urlandresult_infomatch the deployed Artifacts API.OpenAPI. The endpoints are added to
scripts/openapi-filter-config.ts, andopenapi.local.jsonplus the generated types are regenerated from the pinnedcloudflare/api-schemascommit. The diff only adds Artifacts paths. Namespace listing ignoreslimit/cursor, as other local list endpoints do.Edge cases
wrangler devsession using the same persistence directory, and the other way round, including while both run. Artifacts storage is opened directly rather than through a shared-storage owner.remoteURL.cfdisposes Miniflare after each command, so unbound namespaces keep advertising the namespace's recorded Git port even when another process holds it, and never rewritegit-port. The returned remote is therefore the URL a running or later dev server serves. Documented as "clone and push while a dev server runs for that state directory."cf --localpublish no events and start no Workflows, even if a dev server with matchingtriggers.eventsis running. Changes made through a dev server's own explorer reach that server's subscriptions. Neither case errors.namespace.json. It is found once a dev server with this version starts with a binding to it (defaultis always available tocf).Release path
The 5.x line ships from
main. The full simulator (wrapped binding, Git sidecar) first shipped in5.20261113.0-beta. Once this is released,cloudflare/cfneeds only its exactminiflarepin bumped (currently5.20261101.0-beta).Separately:
cf artifacts namespaces repos forkhas no flag for the target repo name, because the bodynameclashes with thenamepath parameter.--body '{"name":"..."}'works around it.LOCAL_DEVELOPMENT.mdandREADME.md; a developers.cloudflare.com update can follow oncecfpicks up the release.New
test/plugins/local-explorer/artifacts.spec.ts(9 tests) covers:cf --localoptions on an empty state directorycf-returned remoteThe Artifacts and Local Explorer suites (454 tests), plus core, Workflows, Flagship and shared (187 tests), all pass.
Manually, with this build linked into a copy of
cf: create, list, fork, get, namespaces list, token create/list/revoke and delete all succeed with--local --persist-to /tmp/s.wrangler dev --persist-to /tmp/swith anartifactsbinding ondefaultthen returns the repository fromenv.ARTIFACTS.list(). While it ran, repositories created by either side were visible to the other, and a clone and push through the remote returned bycfupdatedlast_push_at.