JWKS fetches bypass the CA-aware HTTP client on jwx v3.1.0

[danbarr]

Bug description

ToolHive attaches its CA-aware HTTP client (custom CA bundle, private-IP policy, auth token support) at the httprc client level (pkg/auth/token.go:644-658):

httprcClient := httprc.NewClient(httprc.WithHTTPClient(httpClient))  // CA-aware
cache, err := jwk.NewCache(ctx, httprcClient)
...
v.jwksClient.Register(registrationCtx, v.jwksURL)  // no jwk.WithHTTPClient

jwx v3.1.0 changed Cache.Register to inject jwx's own default client as a resource-level option whenever the caller doesn't supply one (jwk/cache.go:179-181):

// If no HTTP client was explicitly provided, use the library's default
// client which includes timeout and redirect protections.
if !hasHTTPClient {
    resourceOptions = append(resourceOptions, httprc.WithHTTPClient(getFetchHTTPClient()))
}

httprc gives the resource-level client precedence at fetch time (resource.go:222-224: use r.httpcl, fall back to the client-level one only when nil). So on jwx >= 3.1.0 every JWKS fetch silently stops using ToolHive's client and goes through a stock http.Client with no custom CA. Against an issuer whose TLS requires the configured CA bundle (self-signed or private CA), the fetch fails TLS verification, the resource never goes ready, and token validation is dead. Combined with the retry deadlock in #6218, one failed first fetch pins the validator dead for the life of the process.

OSS as shipped is not affected: v0.41.0 and v0.42.0 pin jwx v3.0.13, which sets no resource-level client, so fetches inherit the CA-aware client. But any dependency bump to jwx >= 3.1.0 (e.g. via renovate) breaks JWKS fetching for every deployment using caBundleRef/CACertPath with an external OIDC issuer. Public IdPs with WebPKI certs keep working, which makes the regression easy to miss in CI. The enterprise vmcp image 0.11.0 already bundles jwx v3.1.0 and is broken today in this scenario.

Steps to reproduce

1. Build with jwx >= 3.1.0 (or use enterprise vmcp 0.11.0).
2. Run a VirtualMCPServer with incomingAuth.type: oidc against an external issuer whose TLS cert requires the configured CA bundle (e.g. Keycloak with a self-signed cert, caBundleRef set).
3. Send any request with a Bearer token.

Expected behavior

The JWKS fetch uses the same CA-aware client as OIDC discovery, verifies the issuer's TLS with the configured bundle, and token validation works, as it does on jwx v3.0.13.

Actual behavior

The JWKS fetch uses jwx's stock default client, TLS verification fails against the private CA, the resource never goes ready, and every request fails with:

Invalid token: failed to parse token: token is unverifiable: error while executing keyfunc:
JWKS registration failed: failed to register JWKS URL: failed to add resource to httprc.Client:
resource registered but not ready: context deadline exceeded

The underlying TLS error is swallowed inside httprc's background fetcher, which is what makes this painful to diagnose. Notably, OIDC discovery succeeds in the same pod because discoverOIDCConfiguration uses ToolHive's client directly; only the JWKS path goes through jwk.Cache. That asymmetry (discovery succeeds, JWKS times out) is the diagnostic signature of this bug.

Environment

• Reproduced on stacklok-enterprise/vmcp:0.11.0 (bundles jwx v3.1.0, reports github.com/stacklok/toolhive (devel)), Kubernetes (kind), operator-managed VirtualMCPServer with incomingAuth.type: oidc, external Keycloak issuer, self-signed CA via caBundleRef.
• OSS ghcr.io/stacklok/toolhive/vmcp:v0.41.0 (jwx v3.0.13) works in the identical scenario.

Additional context

Suggested fix: pass the CA-aware client per-resource at registration time:

v.jwksClient.Register(registrationCtx, v.jwksURL, jwk.WithHTTPClient(v.client))

jwk.WithHTTPClient maps to the resource-level httprc option in both v3.0.13 and v3.1.0, so this is a no-op behavior change on the current pin and immunizes the code against the bump. Worth also logging Lookup failures somewhere visible, since httprc hides the underlying fetch error behind ErrNotReady.