[Bug]: Output includes tokens from other prompts when host kv cache enabled

@eopXD

System Info

CPU architecture (e.g., x86_64, aarch64): x86_64
CPU/Host memory size (if known): 2T
GPU properties: NVIDIA H100 and NVIDIA B200
Docker image: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1
(Also tried 2aade46)
NVIDIA driver version

H100 System:
| NVIDIA-SMI 535.154.05 Driver Version: 535.154.05 CUDA Version: 12.2 |

B200 System:
| NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 |

OS (Ubuntu 24.04, CentOS 8): Ubuntu 22.04 (H100) and 24.04 (B200)

Who can help?

This is a follow-up to issue #8274.
This issue was easier to reproduce prior to nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1, but it still happens after the fix.

Information

The official example scripts
My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
My own task or dataset (give details below)

Reproduction

docker run -it --rm --shm-size=250g --gpus "device=4" -p 8000:8000 --entrypoint /bin/bash -v /data:/data nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1
Use the following extra_llm.yaml

enable_chunked_prefill: true
cuda_graph_config:
    max_batch_size: 60
enable_attention_dp: false
kv_cache_config:
    free_gpu_memory_fraction: 0.85
    host_cache_size: 2000000000

trtllm-serve nvidia/Meta-Llama-3.1-8B-Instruct-FP8 --tp_size=1 --backend=pytorch --host=0.0.0.0 --port=8000 --max_seq_len=131072 --max_batch_size=60 --max_num_tokens=2048 --extra_llm_api_options extra_llm.yaml
Open two terminal windows or screen. In the first window, do this:
pkill -f curl
for i in {0..10000}; do ( curl -Z -s http://localhost:8000/v1/chat/completions'?[0-5]' -H "Content-Type: application/json" -d '{"model": "none", "messages": [{"role":"system","content":"Be a helpful assistant #'$RANDOM'. Respond concisely in one sentence."}, {"role": "user","content": "The quick brown fox jumps"}],"reasoning_effort":"high","response_format":{"type":"text"},"temperature":0,"top_k":1,"max_tokens":100,"n":1}' | tr } '\n' | grep --color=never -o 'content":".*' & ); sleep 0.51; done
This will spam a bunch of curl processes. Make sure to pkill -f curl between tests as some get stuck.
In the second terminal window, use the following python script and txt file. Tested with pip install --user httpx==0.28.1

client_bench.py
war_and_peace_excerpt.txt

Run python client_bench.py - it prints lots of httpx.ReadError errors. The exceptions will be ignored, and this is normal. It will create many cancelled requests, but this is fine and may help reproduce the error.
6. Keep both commands running for a while. It may take 5-10 minutes to reproduce the error. Since version 1.2.0rc1, the error happens far less frequently and usually only a few bad responses will show up. When done, make sure to pkill -f curl
7. Test again with host_cache_size: 0. There will be no bad responses, no matter how many times the test is run.

Expected behavior

The responses should all be a sentence, something like this:

content":"Over the lazy dog, which is a well-known pangram sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"That's a well-known pangram, a sentence that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"That is a well-known pangram, a sentence that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The sentence \"The quick brown fox jumps\" is a well-known pangram, a phrase that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog, which is a well-known pangram sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"Over the lazy dog, which is a well-known pangram sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The sentence \"The quick brown fox jumps\" is a well-known pangram, a phrase that uses all the letters of the alphabet at least once.","reasoning_content":null,"reasoning":null,"tool_calls":[]

There should be no repeated words or # or repeated whitespace characters.

actual behavior

The prompt in the above testcase is "Be a helpful assistant #'$RANDOM'. Respond concisely in one sentence. ... The quick brown fox jumps" - so seeing these words verbatim in the output is unexpected.

Most responses are good. Bad responses will include mostly or entirely words from the prompt + chat template like some combination of the following:

content":"#     assistant#   assistant# assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#assistant#","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#             #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#        ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#       ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#       ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#  #   quickbrownfoxjumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"assistant #quick brown fox jumps","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#2661.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#2661.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#26601.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#26601.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#26601.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#2661. Respond concisely in one sentence.","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox jumps\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#           #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #  #","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#      ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"#       ","reasoning_content":null,"reasoning":null,"tool_calls":[]
content":"The quick brown fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox fox","reasoning_content":null,"reasoning":null,"tool_calls":[]

Note in particular the above: My script picks a random number "You are assistant #12345" and performs 5 requests per random number. If there are 6 responses that include the same assistant number, this is proof that prompts are being mixed up across multiple requests, which is a security issue.

I think in general it seems like when the issue happens, it includes a subset of prompt tokens, so running some string comparison against the prompt might be a way to automate detecting this bug.

additional notes

This has been very difficult to get a reproduction for, but it has been an issue for a long time, at least August, probably longer. The issue is also confirmed in 1.1.0x and so on.

Before submitting a new issue...

Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

@pathorn

Thank you for the clear bug issue @pathorn.

Regarding your configuration of the KV cache manager, do you have both enable_block_reuse and enable_partial_reuse enabled? If partial reuse is enabled, may you try disabling it and see if the problem goes away? Thank you.

@eopXD

Thank you, @eopXD
The default for enable_partial_reuse and enable_block_reuse is true, so I was using them.

I added enable_partial_reuse: false to kv_cache_config, and I can confirm at least in my reproduction case, that the issue no longer happens when using host_cache_size.

The issue being related to partial reuse seems consistent with observing bad tokens from the last partial block of the prompt.

@pathorn

I've also recently discovered that a related issue can still occur, despite the fix. It seems to be slightly different this time.

In 1.2.0rc0 the issue was that the LLM was using context tokens from other prompts in the dataset when host_cache_size is enabled.
But in 1.2.0rc1 the issue I'm getting seems to be that the LLM only sees the last few tokens of some prompts. E.g. if the prompt is 1000 tokens long, then it seems as though sometimes the LLM only sees the last 50 tokens. So the tokens that it generates aren't based on other prompts, but are instead based on an incomplete view of the prompt.

I'm not certain about this, but it definitely seems like that's what's happening - very noticably different issue to the one in 1.2.0rc0.

At low host_cache_size values like @pathorn is using, the issue goes away, but at higher values (like 64GB) the issue exists.

The issue seems to only occur under very high concurrency (such that each of the concurrent requests is receiving about 1 token per second).

Here's my extra_llm_api_options config:

stream_interval: 2
cuda_graph_config: null
enable_attention_dp: false
kv_cache_config:
  enable_partial_reuse: false
  enable_block_reuse: true
  dtype: fp8
  free_gpu_memory_fraction: 0.85
  host_cache_size: 68719476736

And here's my start command (running on 4xB200):

docker run -d --shm-size=8g --ulimit memlock=-1 --ulimit stack=67108864 --gpus '"device=0,1,2,3"' -p 4000:4000 -v /root/data:/root/data -w /root --log-opt max-size=10m nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc1 bash -c "trtllm-serve /root/data/nvidia___DeepSeek-R1-0528-FP4 --port 4000 --host 0.0.0.0 --backend pytorch --max_seq_len 9216 --max_batch_size 2048 --max_num_tokens 32768 --tp_size 4 --trust_remote_code --extra_llm_api_options /root/trtllm-config.yml"

And here are the things I've tried, based on the above config (where ❌ means it did not fix it, and ✅ means it seems to have fixed it):

❌ enable_partial_reuse: true
❌ removed dtype: fp8
❌ enable_attention_dp: false
❌ enable_chunked_prefill: true
❌ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
❌ PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync (this seems to cause a memory leak, eventually consuming all system RAM and crashing the server. unsure if this is related to this issue)
❌ PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:8192
✅ reduce host_cache_size from 64GB to 1GB (but such a small host_cache_size defeats the purpose for my use case. also, perhaps this just makes the issue proportionally rare to reproduce)
❌ add --ep_size 4
❌ add moe_config.backend: TRTLLM

@eopXD Unfortunately I haven't yet been able to create a minimal repro, but for now please feel free to let me know some tests to try, if you have any ideas.

@josephrocca

@josephrocca The default for enable_partial_reuse is already true.

Are you sure you tried setting enable_partial_reuse: false in the kv_cache_config? That seems to solve the issue I am seeing. Can you try again with the initial configuration and only add enable_partial_reuse: false

reduce host_cache_size from 64GB to 1GB (

Yeah I am also unable to reproduce the issue, with extremely small host cache sizes. I can reproduce it with 2GB, but then I typed too few 0's and did 200MB instead of 2GB and that does not reproduce the issue.

My guess is smaller amount of host kv cache just makes the problem significantly less likely to reproduce but not impossible, and we just haven't hit the exact conditions to observe it. Plus, such a small amount of host kv cache is not useful.

Are you sure you tried setting enable_partial_reuse: false in the kv_cache_config?

Yes.

My guess is smaller amount of host kv cache just makes the problem significantly less likely to reproduce but not impossible

Yep, this is my guess too. There does seem to be a trend of larger size = faster to hit the issue.

Note though that your issue seems slightly different to mine. In my case it's not about prompt "mixing" (which I did observe in 1.2.0rc0). Instead, my issue with 1.2.0rc1 seems to be that sometimes prompts are being truncated so that only the last few tokens are visible to the LLM when it's generating the response. I haven't yet observed any "wrong prompt/context retrieved" type outputs in 1.2.0rc1 - only "missing prompt/context".

@eopXD

@eopXD This change (by Codex) seems to fix the issue for me:

• Edited ~/TensorRT-LLM/cpp/tensorrt_llm/batch_manager/kvCacheTransferManager.cpp (+4 -0)
    238      copyBlock(offloadBlock, block, pools, false, numTokensToCopy, mode, directory);
    239 +    // Ensure the primary compute stream does not consume the block before the host->device copy finishes.
    240 +    tr::CudaEvent onboardEvent;
    241 +    mOnboardManager.getStream().record(onboardEvent);
    242 +    mBufferManager.getStream().wait(onboardEvent);
    243  }

I cloned the repo at v1.2.0rc1, made that change, built it, and am now not able to replicate the bug, even after several runs of my client script (previously I hit the bug on every client run without fail).

Unsure if this is the correct or most performant way to fix the issue though. Still playing around with it - will update this thread if I learn anything new.

@pathorn

@pathorn : The issue being related to partial reuse seems consistent with observing bad tokens from the last partial block of the prompt.

Yes, I am seeing partial reuse causing the problem and investigating on it now. Will keep you and @josephrocca in-touch on fixing this.

@josephrocca: In 1.2.0rc0 #8274 was that the LLM was using context tokens from other prompts in the dataset when host_cache_size is enabled.

Your observation is correct. The offloading mechanism was flawed, which I fixed while supporting for SWA in the KV cache manager.

@josephrocca: But in 1.2.0rc1 the issue I'm getting seems to be that the LLM only sees the last few tokens of some prompts. E.g. if the prompt is 1000 tokens long, then it seems as though sometimes the LLM only sees the last 50 tokens. So the tokens that it generates aren't based on other prompts, but are instead based on an incomplete view of the prompt.

I think this depends on whether you have set the correct setting of the KV cache manager depending on the model you are running on. In 1.2.0rc1, sliding window attention is supported in the KV cache manager. What you have described seems like the behavior of running sliding window attention. May you double check you are setting the KV cache manager configuration correctly?

For example, for model like GPT-OSS with hybrid attention layers (18 full+18 SWA), setting for the KV cache manager should have:

kv_cache_config:
  max_attention_window: [128, 4096]

For Llama4 which is of all full attention layers, setting for the KV cache manager should have (setting the singular window to something big):

kv_cache_config:
  max_attention_window: [32768]

@eopXD

@eopXD As shown in my repro code above I'm using DeepSeek R1 0528, which I believe doesn't use SWA?

@josephrocca

@josephrocca You are right, let me try reproduce your situation locally.

@eopXD

@eopXD FWIW, this was an alternative fix by Codex that seems to have fixed the issue:

https://gist.github.com/josephrocca/28698c417c31c9fa13d0dc9cd21ea774

(The previous fix is included in that patch, but it's commented out. Codex gave this alternate solution when I asked about increasing performance/throughput.)

But again, I'm not experienced enough here to know whether this is a correct/performant solution to the issue. All I can say for sure is that after that fix I am no longer able to reproduce the bug.

@josephrocca

@josephrocca The fix in gist is very concise after doing some trace. Let me fast track it with you as author to fix this. We also have a fix on-going for partial reuse.

[NVIDIA#8813][fix] Add missing event for block onboard for the kv cac…

@josephrocca

@josephrocca @eopXD The fix offered by Codex is what we already do in KVCacheTransferManager::syncTransfers

https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/executor/executorImpl.cpp#L2362
  https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/executor/executorImpl.cpp#L1790
    https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.cpp#L1068
      https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/batch_manager/allocateKvCache.cpp#L84
        KVCacheManager::refreshBlocks
          BlockManager::refreshBlocks
            WindowBlockManager::refreshBlocks
              KVCacheTransferManager::syncTransfers

For the c++ executor we are already calling syncTransfers, which records the required events on buffer manager stream. It is possible that we missed this in the Python executor. If that is the case, then the proper fix for this issue is to call KVCacheManager::refreshBlocks after the forward function in the Python executor. Note that refreshBlocks does more than record events on onoard and offload streams.

It does not look like the Python executor calls refreshBlocks. Are you seeing this issue with the C++ or Python executor?

@Tabrizian

@Tabrizian Provided a PR that calls refreshBlocks in the Python executor. @josephrocca can you check if this fixes your issue?

#8890

@thorjohnsen

Hi @thorjohnsen I just tested that PR, and unfortunately the issue still occurs. Here are the exact clone/install/serve commands I used on a 4xB200:

apt-get update && apt-get -y install git git-lfs
git lfs install
cd /root
git clone --branch=user/imant/fixRefreshBlocks https://github.com/Tabrizian/TensorRT-LLM
cd /root/TensorRT-LLM
git submodule update --init --recursive
git lfs pull
python3 ./scripts/build_wheel.py
pip install -e .
cat >/root/trtllm-config.yml<<EOF
stream_interval: 2
cuda_graph_config: null
enable_attention_dp: false
kv_cache_config:
  enable_partial_reuse: false
  enable_block_reuse: true
  dtype: fp8
  free_gpu_memory_fraction: 0.85
  host_cache_size: 134586092748
EOF
trtllm-serve nvidia/DeepSeek-R1-0528-FP4 --backend pytorch --max_seq_len 9216 --max_batch_size 2048 --max_num_tokens 32768 --tp_size 4 --trust_remote_code --extra_llm_api_options /root/trtllm-config.yml

I've been seeing this problem in my setup too (although the symptoms are slightly different). Happens way more often since 1.2.0rc0, doesn't happen much under 1.1.0rc5 with enable_partial_reuse: false
I think a commit in 1.2.0rc0 has made this way worse, potentially SWA kv reuse?

I have tried 1.2.0rc1 and also the refreshblocks PR, both doesn't help.

@josephrocca

@josephrocca The PR from @Tabrizian was not enough to fix the synchronization issue, there were multiple bugs in the transfer manager. I have bundled up the necessary changes in this PR #9056. It seems to fix the problem for me. Can you please check if this also works for you?

@josephrocca

@josephrocca , @dmtri35 , @pathorn ,
Would you be able to try the PR #9056 to see if the problem persists?

@thorjohnsen

I did, didn't fix my problem. Also talked to @thorjohnsen and he seems to think that my problem is different from what the PR is trying to address

@dmtri35

From the discussion I had with @dmtri35 and Pankaj Gupta it sounded like your issue happens even when offloading and partial reuse is disabled, which completely eliminates kv cache transfer manager as the culprit. Please correct me if I misunderstood this.

@dmtri35

@dmtri35, this sounds like a separate issue. Could you please open a new ticket with detailed steps to reproduce the problem, including the model, command, version, and any relevant logs?

@josephrocca, @pathorn , if you haven’t already, it would be great if you could try out PR #9056 so we can hopefully close the loop!

Thank you!

@karljang

@karljang @thorjohnsen Sorry for the delay, just tested that PR and can confirm that it fixes the issues I was having - thanks! 🙏

Oh, nice. Thank you for your feedback!!!

Navigation Menu

Repository navigation

[Bug]: Output includes tokens from other prompts when host kv cache enabled #8813

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior

actual behavior

additional notes

Before submitting a new issue...

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Output includes tokens from other prompts when host kv cache enabled #8813

Description

System Info

Who can help?

Information

Tasks

Reproduction

Expected behavior

actual behavior

additional notes

Before submitting a new issue...

Activity

eopXD commented on Oct 31, 2025

pathorn commented on Oct 31, 2025

josephrocca commented on Oct 31, 2025

pathorn commented on Oct 31, 2025

josephrocca commented on Oct 31, 2025

josephrocca commented on Nov 2, 2025

eopXD commented on Nov 3, 2025

josephrocca commented on Nov 3, 2025

eopXD commented on Nov 3, 2025

josephrocca commented on Nov 3, 2025

eopXD commented on Nov 3, 2025

thorjohnsen commented on Nov 3, 2025

thorjohnsen commented on Nov 3, 2025

thorjohnsen commented on Nov 3, 2025

josephrocca commented on Nov 4, 2025

dmtri35 commented on Nov 7, 2025

thorjohnsen commented on Nov 11, 2025

karljang commented on Nov 17, 2025

dmtri35 commented on Nov 17, 2025

thorjohnsen commented on Nov 17, 2025

karljang commented on Nov 17, 2025

josephrocca commented on Nov 17, 2025

karljang commented on Nov 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions