Skip to content

perf(rover-server): stop resolving secrets when listing rovers - #870

Merged
ron96g merged 1 commit into
mainfrom
agents/request-throttling-investigation
Nov 18, 2026
Merged

ron96g merged 1 commit into
mainfrom
agents/request-throttling-investigation

Conversation

@ron96g

@ron96g ron96g commented Nov 17, 2026

Copy link
Copy Markdown
Member

Problem

A client sending ~400 rpm to rover-server caused ~40k rpm against the secret-manager.

GET /rovers read through RoverSecretStore, which resolves every secret reference of every listed Rover. Each reference is one HTTP GET to the secret-manager, made one at a time, with no cache. Rovers have up to 19 secret paths (m2m clientSecret, clientKey, refreshToken and password on API and AI subscriptions and exposures). A full page of 20 Rovers with a few m2m credentials each comes to about 100 secret-manager calls per request.

Change

  • pkg/store: a new RoverRedactedStore wraps the Rover store with the existing obfuscator and the same secret paths. Secret fields come back as ********** and the secret-manager is never called.
  • GetAll (GET /rovers) lists through the redacted store. 0 secret-manager calls per list request (previously one per secret per rover).
  • GET /rovers/{roverId} and the info endpoints are unchanged and still return resolved values.
  • The getAllRovers OpenAPI description documents the redaction and points to GET /rovers/{roverId} for the values. server.gen.go was regenerated with the oapi-codegen version pinned in go.mod (v2.7.1); the diff is the re-encoded embedded spec plus the generator header.

Why not use the plain RoverStore?

The plain store holds the secret references ($<env:team:app:path:checksum>). The checksum is the first 6 bytes of an unsalted SHA-256 of the secret value, so anyone who can list could test password guesses offline. Callers with an obfuscated scope would also get these references instead of **********.

Why a separate store and not the obfuscated access scope?

That scope is a security setting tied to the client's token. This change is a performance decision for one endpoint, so it gets its own store and leaves the scope-based behaviour in common-server unchanged.

Behavior change ⚠️

GET /rovers (and therefore rover-ctl list output) no longer returns plaintext m2m secrets; they are **********. Anyone who needs the values has to request the Rover by id. A Rover without configured secrets is returned unchanged: the redaction only replaces secret fields that are present, and basic-auth password is required by the CRD whenever basic auth is set.

Tests

  • rover_pagination_test.go: GetAll against a Rover with a subscription client-secret reference and an exposure basic-auth password reference. Both come back as **********, the client ID and username are kept, no reference appears in the response, and the resolving store is never called. The test fails if GetAll lists through SecretStore again (checked).
  • stores_secret_test.go: the redacted store redacts every Rover secret path, even for a caller with full access.
  • make test in rover-server passes.

Not in this PR

The info endpoints (/rovers/info, /rovers/{id}/info) still resolve secrets per item; ?names= allows up to 100 names, which comes to up to 200 secret-manager calls per request. A follow-up could cache resolved values by secret reference; that's safe because references change whenever the value changes.

GET /rovers resolved every secret reference of every listed Rover against
the secret-manager, one sequential request per secret field. A full page of
20 Rovers with a few m2m credentials each turned one request into ~100
secret-manager calls (observed: 400 rpm -> 40k rpm).

The list now reads through a redacted store that replaces secret fields
with "**********" without calling the secret-manager. GET /rovers/{id} and
the info endpoints still return resolved values. Returning the raw
secret references instead is not an option: they embed a checksum of the
secret value.

The getAllRovers OpenAPI description documents the redaction and points to
GET /rovers/{roverId} for the secret values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ron96g ron96g self-assigned this Nov 17, 2026
@ron96g
ron96g merged commit 50a5f62 into main Nov 18, 2026
17 checks passed
@ron96g
ron96g deleted the agents/request-throttling-investigation branch November 18, 2026 10:02
dtit-semantic-release Bot pushed a commit that referenced this pull request Nov 18, 2026
## [0.48.3](v0.48.2...v0.48.3) (2026-11-18)

### Bug Fixes

* **api,event,gateway:** preserve encoded upstream paths in gateway routes ([#837](#837)) ([551f3be](551f3be))
* **discovery-server:** expose icto and psiid of applications ([#858](#858)) ([1237c46](1237c46))
* **rover,event:** additional publishers get a client ([#872](#872)) ([af3d8e2](af3d8e2))

### Performance Improvements

* **rover-server:** stop resolving secrets when listing rovers ([#870](#870)) ([50a5f62](50a5f62))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants