Skip to content

[#18609][fix] Do not require encoder tokens for cross prefix reuse #19450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
100milliongold wants to merge 1 commit into
base: main
Choose a base branch
from
+13 −3
Open

[#18609][fix] Do not require encoder tokens for cross prefix reuse #19450

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions cpp/tensorrt_llm/batch_manager/capacityScheduler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,14 @@ prefillWithChunkedContextsAlreadyExecuting(RequestList const& activeRequests,
newlyContributedContextBlocks.insert(summary.firstNewBlock.value());
}
}
if (crossKvCacheManager && crossKvCacheManager->isEnableBlockReuse())
// A features-based encoder (audio enc-dec models such as Whisper) carries no
// encoder tokens, so mEncoderUniqueTokens stays unset: it is derived from
// mEncoderTokens alone. Prefix reuse matches on token prefixes, so there is
// nothing for analyzePrefixReuse to walk there. Leave the summary unset, the
// state both consumers already handle, rather than dereferencing an empty
// optional.
if (crossKvCacheManager && crossKvCacheManager->isEnableBlockReuse()
&& req->getEncoderUniqueTokens().has_value())
{
auto uniqueTokens = *(req->getEncoderUniqueTokens().value());
auto summary = crossKvCacheManager->analyzePrefixReuse(uniqueTokens, *req);
Expand Down Expand Up @@ -341,15 +348,18 @@ std::tuple<RequestVector, RequestVector> GuaranteedNoEvictScheduler::impl(
auto uniqueTokens = req->getUniqueTokens(0);
summary = kvCacheManager.analyzePrefixReuse(uniqueTokens, *req);
}
// See the note on the same dereference above.
if (crossKvCacheManager && crossKvCacheManager->isEnableBlockReuse()
&& !crossKvCacheManager->getBlockManager().isVariableWindow())
&& !crossKvCacheManager->getBlockManager().isVariableWindow()
&& req->getEncoderUniqueTokens().has_value())
{
auto uniqueTokens = *(req->getEncoderUniqueTokens().value());
crossSummary = crossKvCacheManager->analyzePrefixReuse(uniqueTokens, *req);
}
}
else if (isEncoderInit && crossKvCacheManager && crossKvCacheManager->isEnableBlockReuse()
&& !crossKvCacheManager->getBlockManager().isVariableWindow())
&& !crossKvCacheManager->getBlockManager().isVariableWindow()
&& req->getEncoderUniqueTokens().has_value())
{
// Encoder admission only needs the cross summary for reuse ordering.
auto uniqueTokens = *(req->getEncoderUniqueTokens().value());
Expand Down