feat(llm): add public api_key property for in-place credential rotation - #2300
Open
waitasecant wants to merge 1 commit into
Open
feat(llm): add public api_key property for in-place credential rotation#2300waitasecant wants to merge 1 commit into
waitasecant wants to merge 1 commit into
Conversation
github-actions
Bot
added
size: M
status: needs triage
Callers that hold one long-lived LLMRails/OpenAIChatModel instance and authenticate with a rotating bearer token had no supported way to update it in place, short of reaching into private attributes or leaking a client via DefaultFramework's api_key-keyed cache. - Add BaseClient.api_key getter/setter (llm/clients/base.py). - Delegate through OpenAIChatModel.api_key (llm/models/openai_chat.py). - Forward through InstrumentedLLMModel.api_key via getattr/setattr so tracing/metrics wrapping doesn't silently no-op rotation (llm/models/instrumented.py). - Document api_key as an optional, bearer-token-only extension on the LLMModel protocol, not a required member (types.py). - Add tests covering getter/setter round-trip, header propagation, and delegation through the instrumentation decorator. Purely additive; no existing signature or behavior changes. Signed-off-by: Himanshu waitasecant@gmail.com
waitasecant
force-pushed
the
feat/llm-client
branch
from
August 18, 2026 07:55
096b740 to
ae4aaa3
Compare
Author
|
Hi @Pouyanpi @tgasser-nv @miyoungc could you look into this PR put out an update Friendly ping on this PR 🙂 This is a small, additive change that exposes a public The PR has been open for about 3 weeks and appears to still be awaiting triage/review. When you have a chance, could you please take a look and/or approve the pending workflows so CI can run? Thanks for your time! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Applications that build one long-lived
LLMRails/OpenAIChatModelinstance at process startup (a common pattern for services that don't want per-request construction overhead) and authenticate against their LLM provider with a short-lived, rotating bearer token (OAuth client-credentials, AWS STS-style tokens, internal gateway tokens, etc.) currently have no supported way to update that credential in place.Today the only two options are:
rails.llm._client._api_key), which is unsupported and can silently break on any minor/patch upgrade with no type or import error — just silent 401s from the provider.DefaultFramework.create_model()and swap it in withLLMRails.update_llm(). This "looks" like the supported path, butDefaultFramework._clientscachesOpenAICompatibleClientinstances keyed by(base_url, api_key, ...)and never evicts entries, so every token rotation leaks a newhttpx.AsyncClientconnection pool for the life of the process.This PR adds a small, purely additive
api_keyproperty (with setter) so callers have a public, documented way to rotate the credential on an already-built model without either workaround.Related Issue(s)
api_keyproperty for in-place LLM credential rotation #2299Verification
Ran pytest and pre-commit checks successfully.
AI Assistance
Checklist