feat(server): add credential storage drivers for Provider v2

[varshaprasad96]

Summary

Add pluggable credential storage drivers so provider secrets are stored in external backends instead of inline in the gateway database. This implements the credential storage driver system described in issue #1931.

Three backends are supported:

• Default encrypted DB credstore — AES-256-GCM envelope encryption with two-layer key wrapping (per-credential DEK wrapped by a KEK), stored in the gateway's existing object store
• Kubernetes Secrets — credentials stored as Opaque Secrets in a configurable namespace with ownership validation
• Vault — credentials stored in a Vault-compatible KV v1/v2 backend with Kubernetes or token-file authentication

Key design decisions

• Opaque credential handles: Provider records store credential_handles (driver + handle reference) instead of inline secrets. API responses redact both inline and handle-based credentials.
• CAS (Compare-And-Swap) lifecycle: Credentials are pre-stored externally before the CAS write to the gateway DB. On CAS failure, best-effort cleanup deletes stored handles with structured orphan warnings.
• Dual-path coexistence: Provider.credentials (inline) and Provider.credential_handles (driver-managed) coexist for backward compatibility. Existing inline credentials remain readable; new operations use handles when a driver is configured.
• Single driver enforcement: At most one external credential driver can be enabled at a time, validated at both Helm and runtime levels.

Improvements over PR #1986

This implementation is based on PR #1986 by @TaylorMutch with the following improvements:

1. Parallel RPC dispatch — store_provider_credentials and delete_provider_credential_handles use futures::future::join_all instead of sequential loops
2. Vault auth token caching — RwLock-based double-checked locking with TTL from lease_duration, avoiding re-authentication on every RPC
3. Namespace enforcement — allow_reference_namespace is now actually enforced at runtime in store/delete/resolve paths (was parsed but never checked)
4. Additional credentials routing — apply_minted_credential routes additional_credentials (e.g. AWS STS session tokens) through the driver instead of writing them inline as plaintext
5. CAS failure cleanup in refresh — Stored handles are cleaned up on CAS failure in apply_minted_credential instead of being orphaned
6. Enriched orphan warnings — Structured logging with handle_count, per-handle (key, driver, handle) tuples, and provider name
7. RBAC documentation — YAML comment on the Role template and security note in docs explaining broad secrets access and dedicated namespace recommendation
8. GitOps KEK support — defaultKeyEncryptionKey Helm value for helm template workflows where lookup/randBytes are unavailable

Related Issue

Closes #1931

Changes

New crates

• crates/openshell-driver-db-credstore/ — Default encrypted DB credential store (AES-256-GCM)
• crates/openshell-driver-kubernetes-secrets/ — Kubernetes Secrets credential driver
• crates/openshell-driver-vault/ — Vault credential driver

Server

• crates/openshell-server/src/credentials.rs — Credential runtime, driver registry, parallel RPC dispatch
• crates/openshell-server/src/grpc/provider.rs — CAS-aware credential handle lifecycle in create/update/delete
• crates/openshell-server/src/provider_refresh.rs — Route refreshed credentials through driver with CAS cleanup
• crates/openshell-server/src/inference.rs — Resolve credential handles for inference routes
• crates/openshell-server/src/grpc/validation.rs — Credential handle field validation

Helm

• KEK Secret template with lookup > user-supplied value > randBytes priority
• Kubernetes Secrets RBAC (Role + RoleBinding) with namespace scoping
• Gateway config rendering for all three driver backends
• Helm unit tests for all configurations

Docs

• docs/reference/gateway-config.mdx — Full credential drivers reference section
• docs/sandboxes/providers-v2.mdx — User-facing credential storage documentation
• architecture/gateway.md — Architecture documentation

E2E

• e2e/rust/tests/credential_drivers.rs — End-to-end tests for K8s Secrets and Vault backends
• CI workflow integration with test:e2e-kubernetes label

Testing

• mise run pre-commit — lint, format, clippy all pass
• mise run test — all unit tests pass (including new credential driver tests)
• mise run helm:test — Helm unit tests pass (9 credential driver test cases)
• test:e2e-kubernetes — E2E tests for K8s Secrets and Vault backends (CI)

Checklist

• Conventional Commits format
• Signed-off commits (DCO)
• No secrets or credentials committed
• Documentation updated (gateway-config.mdx, providers-v2.mdx, architecture/gateway.md)
• AGENTS.md updated with new crate entries
• CI.md and TESTING.md updated with new e2e test documentation

Co-authored-by: Taylor Mutch taylormutch@gmail.com

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[varshaprasad96]

Leaving this in draft mode, this is the rebased PR with some fixes on top of #1931.

Only one of #2437 or #2454 will be merged, so leaving it in drat mode for now.

@sjenning could you please take a look

[johntmyers]

Is this a dupe of #2437? Can we just pick one?

[varshaprasad96]

Is this a dupe of #2437? Can we just pick one?

Yes, we are reviewing the changes in both, and will pick up. It was journey of huge rebase in 2 different ways. Which is why this is in draft. Will proceed with only one.