[aron-muon]

Summary

#6822 checks a cached upstream DCR registration at startup by redeeming a random authorization code at the token endpoint: invalid_client means the registration is gone, invalid_grant means it is alive. That decides only for upstreams that authenticate the client before validating the grant. A vendor whose token endpoint answers invalid_grant for any client_id (the production case in #6821: dead, live and made-up clients all get 400 invalid_grant "Invalid authorization code") keeps presenting a dropped registration, and the user sees the rejection in the browser while the resolver sees nothing.

• Add dcrConfig.registrationProbe on MCPExternalAuthConfig upstream providers (and the VirtualMCPServer inline copy) and registration_probe on the authserver.DCRUpstreamConfig run-config, with values tokenEndpoint (default, exactly Re-register DCR clients the upstream evicted #6822's behaviour), authorizationEndpoint and none. Enum-validated at admission and mirrored in the run-config Validate and the resolver's input check, so an unknown value fails at startup instead of silently probing the wrong endpoint.
• authorizationEndpoint sends an authorization request for the cached client (response_type=code, client_id, the registered redirect_uri, the configured scopes, a throwaway state and S256 PKCE challenge) and reads the first response without following redirects. RFC 6749 §4.1.2.1 has the server answer an unknown client_id or redirect_uri directly rather than by redirect, so a direct 400, 401 or 403 carrying an RFC 6749 JSON error object with error invalid_client, unauthorized_client or invalid_request means the registration is gone, and the resolver re-registers through the same Replacer.ReplaceIfUnchanged path as the token-endpoint probe. 2xx (a login page) and 3xx (to the identity provider, or back to redirect_uri with an error) mean the client is known. Everything else is inconclusive and keeps the cached registration: other 4xx, a 4xx without a JSON error object (an HTML page, a bare gateway status), a JSON error outside that set, 5xx and transport failures. The request carries no credentials; up to 16 KiB of the body is read for the error code, which is repeated in logs only when it matches ^[a-z_]{1,40}$; the description is never logged. A false rejection re-registers at every start and, when registration also fails, fails startup, which is why the verdict is narrow. Because it sends no credentials it also covers private_key_jwt clients, which the token-endpoint probe skips.
• none skips the check for upstreams that should not see probe traffic; the registration is then retired only through rotationKey.
• The configured authorization endpoint wins over the cached one, as the token probe does, so a tampered row cannot redirect the probe. Same guarded HTTP client, timeout and SSRF posture.
• rotationKey docs now say the rejection "may not be visible to the resolver" and point at the probe setting; the cache-hit log records carry registration_probe.

Related to #6821. Builds on #6822.

Type of change

• Bug fix
• New feature
• Refactoring (no behavior change)
• Dependency update
• Documentation
• Other (describe):

Test plan

• Unit tests (task test) — every touched package passes with -race (pkg/auth/dcr, pkg/authserver, pkg/authserver/storage, pkg/authserver/runner, cmd/thv-operator/pkg/controllerutil, cmd/thv-operator/api/...). New coverage in pkg/auth/dcr/validate_test.go:
  • TestResolveCredentials_CacheHitAuthorizationEndpointProbe: 200, 302 to the IdP, 302 back to redirect_uri with an error, 400 invalid_request, 401 invalid_client, 403 unauthorized_client, 403 HTML page, bodiless gateway 401, 400 server_error, 400 non-error JSON, 404, 405, 429, 500, failed re-registration; asserts the request shape, that no credentials are sent, that the token endpoint is never called and that redirects are not followed.
  • ..._AuthorizationProbeVerifiesPrivateKeyJWTClients, ..._CacheHitProbesConfiguredAuthorizationEndpoint (decoy cached endpoint), ..._AuthorizationProbeTransportErrorKeepsRegistration, ..._ProbeNoneSkipsUpstream.
  • TestClassifyAuthorizationProbe (status and body matrix), TestAuthorizationProbeErrorAttrs_OmitsServerResponse, ..._RepeatsOnlyTheErrorCode, TestVerifyCachedRegistration_* (rejected probe endpoints, redirect-URI fallback).
  • The existing token-probe table also pins that it never touches the authorization endpoint. Run-config validation, newDCRRequest propagation and the operator CRD-to-run-config conversion each gained the new field.
• E2E tests (task test-e2e)
• Linting (task lint-fix)
• Manual testing (describe below)

API Compatibility

• This PR does not break the v1beta1 API, OR the api-break-allowed label is applied and the migration guidance is described above.

registrationProbe is a new optional enum string on DCRUpstreamConfig; an absent or empty value keeps #6822's token-endpoint probe. No storage format or Redis key change.

Changes

File	Change
pkg/auth/dcr/request.go	RegistrationProbe type, constants, Validate; Request.RegistrationProbe
pkg/auth/dcr/validate.go	Dispatcher, verifyViaAuthorizationEndpoint, classifyAuthorizationProbe; guarded client builder shared by both probes
pkg/auth/dcr/resolver.go	Reject unknown values in validateResolveInputs; registration_probe log attribute; ProbeNone logs at Debug
pkg/authserver/config.go	registration_probe on the run-config, validated through the resolver's enum
pkg/authserver/runner/dcr_adapter.go	Pass the field to dcr.Request
cmd/thv-operator/api/v1beta1/mcpexternalauthconfig_types.go	registrationProbe with an Enum marker; rotationKey wording
cmd/thv-operator/pkg/controllerutil/authserver.go	Copy the field into the run-config
pkg/authserver/storage/types.go	rotationKey wording
deploy/charts/operator-crds/**, docs/operator/crd-api.md, docs/server/*, sdk/go/*	Generated
docs/arch/11-auth-server-storage.md	Describe the probe selection

Does this introduce a user-facing change?

Yes. Operators can set dcrConfig.registrationProbe: authorizationEndpoint on an upstream whose token endpoint cannot tell a dropped public client from a live one; the proxy then re-registers at its next start instead of presenting the dead client_id until someone rotates it. The default is unchanged. The authorization endpoint sees one abandoned authorization request per upstream per start.

Special notes for reviewers

• The verdict needs the server's own structured answer, not a status alone: a direct 400/401/403 with an RFC 6749 JSON error object in the rejection set. Measured shapes: the Cached upstream DCR registration never recovers after the upstream evicts it #6821 vendor answers an unknown client with 400 {"error":"invalid_request","error_description":"redirect_uri is not in the allowlist..."}, a ToolHive auth server with 401 {"error":"invalid_client", ...}; both decide. Servers that render an HTML page (Keycloak, Auth0), a gateway's 401/403, or a 200/redirect error page read as unverified or alive, which is today's behaviour; rotationKey remains the remedy there.
• none is included because an operator may not want probe traffic against a vendor that logs or alerts on failed requests; it is one case in the dispatcher and the resolver logs it at Debug rather than Warn.
• The two probes share newProbeHTTPClient; each sets its own CheckRedirect (refuse for the token probe, which carries credentials; ErrUseLastResponse for the authorization probe, where the redirect is the answer).

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 95.83333% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.92%. Comparing base (76b09c0) to head (57fdf8c).

Files with missing lines	Patch %	Lines
pkg/auth/dcr/validate.go	95.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6834      +/-   ##
==========================================
- Coverage   79.94%   79.92%   -0.03%     
==========================================
  Files         833      834       +1     
  Lines       84977    85063      +86     
==========================================
+ Hits        67938    67984      +46     
- Misses      17034    17074      +40     
  Partials        5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aron-muon]

@jhrozek checking in — does cf6d46c address the three points above? Happy to iterate further if not.

One more data point in case it's useful for prioritizing: the #6821 vendor just purged its registered-client store a third time (Oct 19, worked around again by deleting the Redis row and restarting). The gap between purges is shrinking — 49 days, then 12, then 5 — so this is becoming a recurring manual fire drill on our side rather than a one-off. Keen to get this merged so the next purge heals itself at the next pod restart instead.

[jhrozek]

I apologize for the long delay @aron-muon - there was too many things going on at once. I fixed the conflict and merged the PR, so it will be out in the next release (we release several times a week these days)