Skip to content

fix(api): server delete takes locks in a deadlock-free order - #46

Merged
MihaiM21 merged 2 commits into
masterfrom
fix/server-delete-lock-order
Nov 20, 2026
Merged

MihaiM21 merged 2 commits into
masterfrom
fix/server-delete-lock-order

Conversation

@MihaiM21

Copy link
Copy Markdown
Owner

What and why

Follow-up to #45. That fix made DELETE /servers/{id} lock the servers row first, then write agent_certificates (revoke) and secrets (credential release) on save. Certificate sync after a renewal (AgentSessionHandler.SyncCertificatesAsync) and the PATCH that changes sshCredentialSecretId take those locks the other way round, so a renewed agent reconnecting while its server is deleted could deadlock: PostgreSQL aborts one side with 40P01, which surfaced as a 500 on the delete or an unhandled error in the sync.

Changes:

  • Lock order in the delete is now secrets, then agent_certificates, then servers, the same order the other writers use. The credential id is read unlocked first and re-checked once the locks are held; if it changed, the delete starts over.
  • Retry: one retry on 40P01/40001, then a 409 with the new problem code server.busy.
  • Org scoping: every lock query filters on organization_id, so a DELETE with another org's server id returns 404 without locking that row (before, it briefly stalled that server's heartbeat writes).
  • Certificate sync also retries when PostgreSQL picks it as a deadlock or serialization victim, as a safety net.

No migration, proto or OpenAPI changes.

How I verified it

With AETHERA_TEST_DB set (Postgres 17):

  • dotnet build -warnaserror: 0 warnings, 0 errors.
  • Server and agent tests (AgentApiTests|Servers): 62 passed, 1 skipped (SSH integration), 0 failed.
  • Two new tests: one where another transaction locks the certificate row and then writes the server row during the delete, and one for another org's server id. Both fail on the old delete code and pass with this change.
  • StaticWebTests has two failures locally that also fail on master: Windows path assertions (D:\repo\... vs /repo/...), unrelated. The full suite on Windows is noisy from ephemeral-port exhaustion; CI is the clean signal.

Checklist

  • Tests added or updated (a bug fix has a test that fails without it)
  • The build and tests pass for what I touched: dotnet build -warnaserror && dotnet test (with AETHERA_TEST_DB set), go vet ./... && go test ./..., pnpm lint && pnpm test && pnpm build
  • No secrets in code, logs, errors or test output
  • CHANGELOG.md updated under Unreleased if users or operators would notice (not updated, same as fix(api): deleting a server no longer races its agent's writes #45: a rare race)
  • Docs updated, and an ADR if this contradicts or extends one (none needed)
  • Conventional Commit messages, focused diff

Deleting a server locked the servers row first and then wrote
agent_certificates and secrets, the reverse of certificate sync and
the credential PATCH, so a concurrent renewal could deadlock it.

The delete now locks secrets, then agent_certificates, then servers,
re-checks the credential once the locks are held, retries one
deadlock or serialization failure and answers 409 server.busy on a
second. Every lock query is scoped to the caller's organization, so
another org's server id takes no lock. Certificate sync also retries
when it is chosen as a deadlock victim.
…n string

The deadlock and cross-org lock tests read the connection string back from an opened DbContext, which Npgsql returns without the password. CI's Postgres requires one, so both tests failed to connect there; use the test factory's connection string instead.
@MihaiM21
MihaiM21 merged commit 055369c into master Nov 20, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant