Skip to content

fix(api): deleting a server no longer races its agent's writes - #45

Merged
MihaiM21 merged 1 commit into
masterfrom
fix/server-delete-conflict
Nov 20, 2026
Merged

MihaiM21 merged 1 commit into
masterfrom
fix/server-delete-conflict

Conversation

@MihaiM21

Copy link
Copy Markdown
Owner

DELETE /api/v1/servers/{id} could answer 409 Conflict to a request that sent no If-Match when the server's agent was connected. The agent persists heartbeats and status on the server row at any moment. A write between the delete's read and its save changed the row version (xmin), so the save failed the concurrency check. It showed up in CI as a flaky Deleting_a_server_revokes_its_certificates_and_closes_its_live_stream.

Fix: the delete locks the server row with SELECT … FOR UPDATE in a transaction before reading it. The agent's writes wait the few milliseconds until the commit, and If-Match is checked against the locked version, so a stale ETag still gets 412.

I tried an optimistic retry first, but it starves under continuous writes (still 409 after 5 attempts), hence the lock.

Tests (AgentApiTests):

  • Deleting_a_server_succeeds_while_its_agent_keeps_writing_the_row: a background writer updates the row non-stop during 5 deletes. Fails on master with 409; passes with the fix, 5 out of 5 runs.
  • Deleting_a_server_with_a_stale_if_match_is_still_refused: 412 and the server isn't deleted.
  • Locally against Postgres 16 and Redis 7: the Agents, Resources, Ssh and Placement test namespaces, 1454 passed, 0 failed.

Known edge: an agent reconnecting with a renewed certificate at the exact moment its server is deleted writes the certificate and server rows in the opposite order. Postgres would detect the deadlock and abort one of the two. That is rarer than the bug fixed here and resolves itself.

🤖 Generated with Claude Code

A connected agent persists heartbeats and status on the server row at any
moment. When one landed between the delete's read and its save, the xmin
check failed and DELETE /servers/{id} answered 409 to a client that sent
no If-Match (seen in CI as a flaky AgentApiTests failure).

The delete now locks the server row (SELECT ... FOR UPDATE) in a
transaction before reading it, so the agent's writes wait for the commit.
If-Match is checked against the locked version, so a stale ETag still
gets 412.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MihaiM21
MihaiM21 merged commit 661e279 into master Nov 20, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant