Skip to content

fix(config): concurrent read-modify-write operations silently lose updates #832

Closed

Description

@PierrunoYT

Version / branch / commit

main at 097c2652cba22d929c598347aacc6f95591c9031

OS and environment

Static persistence review; behavior is cross-platform and requires two Zero processes sharing one config file.

Steps to reproduce

  1. Start two processes against the same config revision.
  2. Process A calls a writer mutation such as UpsertProvider.
  3. Process B concurrently performs a different config mutation.
  4. Both read the same old document, modify independent fields, and atomically rename a complete replacement.
  5. Observe that whichever rename occurs last silently discards the other successful update.

Representative code: internal/config/writer.go:14-55 and internal/config/writer.go:738-772.

The temp-file rename prevents partial JSON, but no cross-process lock or revision check covers the read-modify-write transaction.

Expected behavior

Independent successful config mutations should both survive, or a stale writer should receive a conflict error and retry from the latest revision.

Actual behavior

The final file is valid JSON but one acknowledged update is lost without warning.

Suggested fix and regression test

Hold a cross-process lock across load, mutation, validation, and atomic publication. Re-read after acquiring the lock. Optionally include a generation/revision check to reject stale writes.

Add a coordinated two-process test that mutates different fields and verifies both are present in the final config.

Reported by PierrunoYT from a full codebase audit.

Activity

  1. Vasanthdev2004 commented on Aug 5, 2026

    @Vasanthdev2004
    Collaborator

    Real, and still present. Every writer in internal/config/writer.go is read-modify-write with an atomic rename at the end, and nothing carries a revision, so the last rename wins and the other process's successful update disappears with no error.

    Sorry for the wait. We want it fixed, but not as a drive-by: this needs one mechanism applied to every writer in that file rather than a patch on UpsertProvider, and the choice matters. My preference is a file lock around read-modify-write using the existing internal/lockutil, since it is already the repo's answer to this problem and avoids a retry loop in every caller. If you would rather do compare-and-swap on a revision field, say so and why before you start.

  2. Vasanthdev2004 commented on Aug 12, 2026

    @Vasanthdev2004
    Collaborator

    Approved. Verified at internal/config/writer.go:14.

    UpsertProvider reads the whole document with os.ReadFile, merges into the in-memory copy, and writes a complete document back. There is no file lock, no revision check, and no compare-and-swap, so two processes that read the same revision each write a full document and the second erases the first's unrelated mutation. The atomic rename makes each individual write safe; it does not serialize them, which is the distinction this issue is making.

    Concretely: this repository already has internal/lockutil for exactly this class of problem. Whoever picks this up should look there first rather than inventing a scheme, and should read #831 alongside it, since the same lock protocol has an accepted race worth understanding before relying on it.

  3. added
    issue-approvedReviewed and approved by the core team; community PRs may implement this issue.
    on Aug 12, 2026
  4. gnanam1990 commented on Aug 24, 2026

    @gnanam1990
    Collaborator

    Re-reviewed on current main. Still real, and the measured loss rate is worse than the report implies.

    Worth stating up front because a sibling audit issue did not survive re-review: #907 turned out to be fully fixed by #903 and a dependency bump, so I checked this one against live main rather than assuming.

    Still unguarded

    grep -n "lockutil|securefile|flock|Lock"  internal/config/writer.go   -> no matches
    grep -rln "lockutil|securefile"           internal/config/            -> none
    

    UpsertProvider (writer.go:14-55) still reads with os.ReadFile, mutates in memory, and calls writeConfigFile, with nothing spanning the three. The repo has internal/lockutil and internal/securefile; neither is used here. Every caller invokes it directly — internal/tui/provider_wizard.go:1275, internal/cli/setup.go:269, internal/cli/provider_setup.go:58 — so no caller supplies the missing lock either.

    Measured

    Concurrent UpsertProvider calls, each adding a distinct provider to a shared file:

    concurrent writers providers surviving lost
    2 1 1
    8 1 7
    20 2 18

    And the part that makes it dangerous rather than merely lossy:

    writes acknowledged OK : 10
    writes that errored    :  0
    providers in final file:  1
    -> 9 writes returned success and vanished
    final JSON is valid    : yes
    

    Every lost write returned no error. The atomic rename does its job — the file is never partial — so the result is a valid config that silently contradicts what nine callers were told had been saved. Nothing downstream can detect it, which is why "silently" in the title is the operative word.

    One correction to the report's framing

    It describes this as requiring "two Zero processes sharing one config file". It does not: the numbers above are goroutines inside a single process, because there is no in-process lock either. That widens the reachable surface — a single Zero doing two config mutations concurrently is enough — and it means an in-process mutex would fix the measured case while leaving the cross-process case the report actually describes wide open. The fix needs to be the cross-process one to cover both.

    On the suggested fix

    Holding a lock across load-mutate-validate-publish is right, and internal/securefile already demonstrates the shape in this repo (lock, re-read, write temp, rename) for the credential store — the same asymmetry #921 turns on, where Zero's own secrets get durability guarantees the user's data does not.

    The revision check is worth keeping as more than an "optionally": with a lock alone, a writer that loaded before acquiring still publishes stale state. Re-reading after acquiring, as the report says, is what actually closes it — the generation counter is then belt-and-braces for a caller that holds a FileConfig across the lock boundary.

    The suggested regression should assert both survivors and that no call returned success for a write that vanished, since the acknowledgement is the part that makes this silent.

    Label stays; nothing here changes the approval.

  5. added a commit that references this issue on Sep 12, 2026
    b04fe96
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingissue-approvedReviewed and approved by the core team; community PRs may implement this issue.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions