[https://nvbugs/6658015][fix] Pin uvloop<0.23.0 to restore trtllm-serve streaming throughput by chenfeiz0326 · Pull Request #19993 · NVIDIA/TensorRT-LLM

chenfeiz0326 · 2026-10-08T15:21:48Z

Summary

Cherry-pick of #19992 to release/1.3: cap uvloop below 0.23.0 in requirements.txt.

Fixes https://nvbugs/6658015 on release/1.3.

Only the requirements.txt change applies here. The waives.txt entry that #19992 removes on main does not exist on release/1.3.

Why release/1.3 needs it

requirements.txt on release/1.3 also declares only uvloop>=0.19.0, so CI picks up uvloop 0.23.0 (published 2026-10-01 03:15 UTC). The GB200 perf-sanity case aggr_upload-deepseek_r1_fp4_v2_grace_blackwell-r1_fp4_v2_dep4_mtp1_1k8k regressed on this branch in the same window as on main:

release-1.3 L0_PostMerge Kaiyu/update main #5 (2026-09-30) passed in 74 min.
update aarch64 libraries to main branch #6 onward time out at the 90-minute budget.
The only code commit in the Kaiyu/update main #5 → update aarch64 libraries to main branch #6 range ([https://nvbugs/6713426][fix] Make MoE communication timeouts follow the warmup phase #19593) is not on main. The common cause is the dependency install.

A/B verification (GB200, oci-hsg)

Same L0_PostMerge #3003 wheel and same config on both arms; only the uvloop version differs, confirmed on each node:

	uvloop 0.22.1	uvloop 0.23.0
Benchmark duration	4616 s (76.9 min)	5745 s (95.8 min)
Total token throughput	20443 tok/s	16426 tok/s (−19.6%)
Median TTFT	20.0 s	172.2 s
Median TPOT	104.1 ms	112.3 ms

See #19992 for the full root-cause analysis.

Test plan

Local A/B on GB200 above (main-branch wheel)
/bot run --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1"

🤖 Generated with Claude Code

Dev Engineer Review

requirements.txt now constrains uvloop to >=0.19.0,<0.23.0. The change prevents installation of uvloop 0.23.0 and addresses the reported GB200 performance regression. The reported A/B benchmark took 76.9 minutes with 0.22.1 and 95.8 minutes with 0.23.0. The requested GB200 post-merge test has no reported result.

QA Engineer Review

No test changes.

Per-File QA Perspective

requirements.txt: Verify that dependency installation selects a uvloop version from 0.19.0 up to, but not including, 0.23.0. Confirm the GB200 performance-sanity case stays within its 90-minute budget with the capped dependency.

…ve streaming throughput uvloop 0.23.0 (released 2026-10-01) regresses trtllm-serve throughput at high streaming concurrency, pushing the GB200 perf-sanity case r1_fp4_v2_dep4_mtp1_1k8k past its 90-minute budget. Cap uvloop below 0.23.0. Cherry-picked from main PR NVIDIA#19992 (requirements.txt part only; the waiver removed there is not present on release/1.3). Signed-off-by: chenfeiz0326 <203214996+chenfeiz0326@users.noreply.github.com>

chenfeiz0326 · 2026-10-08T15:22:01Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-PerfSanity-Post-Merge-1"

coderabbitai · 2026-10-08T15:24:34Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: NVIDIA/TensorRT-LLM/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e50ba0bb-c01c-4c45-b2dc-8e3c5e3c5d05

📥 Commits

Reviewing files that changed from the base of the PR and between 1e43218 and 28725d9.

📒 Files selected for processing (1)

requirements.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The uvloop requirement now specifies a lower bound of 0.19.0 and an upper bound below 0.23.0.

Changes

uvloop Requirement

Layer / File(s)	Summary
Set uvloop version bounds `requirements.txt`	The requirement now accepts `uvloop` versions `>=0.19.0` and `<0.23.0`.

Priority: ⬆️ High

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Bug fix

Suggested reviewers: emmaqiaoch

Merge Risk: ⚪ Minimal · up to 28725

The version cap prevents installing the version linked to the reported slowdown. The post-merge benchmark result is still unreported, but no concrete merge-blocking risk is established.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description clearly explains the issue, the release-branch context, the dependency change, A/B verification results, and the test plan. It does not reproduce the full checklist template, but it pr…
Title check		The title identifies the NVBugs issue, uses the valid fix type, names the uvloop constraint, and states the performance objective. It accurately summarizes the primary change.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Create a new PR

Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-10-08T15:29:02Z

PR_Github #76757 [ run ] triggered by Bot. Commit: 28725d9 Link to invocation

tensorrt-cicd · 2026-10-08T16:27:35Z

PR_Github #76757 [ run ] completed with state FAILURE. Commit: 28725d9
/LLM/release-1.3/L0_MergeRequest_PR pipeline #101 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

github-actions · 2026-10-09T05:36:17Z

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

chenfeiz0326 · 2026-10-09T06:24:59Z

/bot run --post-merge --disable-fail-fast

tensorrt-cicd · 2026-10-09T06:31:05Z

PR_Github #76903 [ run ] triggered by Bot. Commit: 28725d9 Link to invocation

tensorrt-cicd · 2026-10-09T06:31:06Z

PR_Github #76903 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

chenfeiz0326 · 2026-10-09T09:49:08Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-09T09:56:48Z

PR_Github #76941 [ run ] triggered by Bot. Commit: 28725d9 Link to invocation

tensorrt-cicd · 2026-10-09T14:09:55Z

PR_Github #76941 [ run ] completed with state SUCCESS. Commit: 28725d9
/LLM/release-1.3/L0_MergeRequest_PR pipeline #127 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-10-09T14:31:50Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-09T14:39:02Z

PR_Github #76967 [ run ] triggered by Bot. Commit: 28725d9 Link to invocation

tensorrt-cicd · 2026-10-09T18:58:35Z

PR_Github #76967 [ run ] completed with state SUCCESS. Commit: 28725d9
/LLM/release-1.3/L0_MergeRequest_PR pipeline #131 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

chenfeiz0326 · 2026-10-10T03:36:02Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-10-10T03:43:07Z

PR_Github #77082 [ run ] triggered by Bot. Commit: 28725d9 Link to invocation

tensorrt-cicd · 2026-10-10T05:12:02Z

PR_Github #77082 [ run ] completed with state SUCCESS. Commit: 28725d9
/LLM/release-1.3/L0_MergeRequest_PR pipeline #147 completed with status: 'SUCCESS'

CI Report

Link to invocation

chenfeiz0326 requested a review from a team as a code owner October 8, 2026 15:21

github-actions Bot assigned chenfeiz0326 Oct 8, 2026

yuanjingx87 approved these changes Oct 9, 2026

View reviewed changes

trtllm-agent added the ci: full pre-merge approved label Oct 9, 2026

chenfeiz0326 enabled auto-merge (squash) October 9, 2026 06:25

chzblych approved these changes Oct 9, 2026

View reviewed changes

chenfeiz0326 merged commit 7aa99f9 into NVIDIA:release/1.3 Oct 10, 2026

Conversation

chenfeiz0326 commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why release/1.3 needs it

A/B verification (GB200, oci-hsg)

Test plan

Dev Engineer Review

QA Engineer Review

Per-File QA Perspective

Uh oh!

chenfeiz0326 commented Oct 8, 2026

Uh oh!

coderabbitai Bot commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

tensorrt-cicd commented Oct 8, 2026

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

chenfeiz0326 commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

chenfeiz0326 commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

chenfeiz0326 commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

tensorrt-cicd commented Oct 9, 2026

Uh oh!

chenfeiz0326 commented Oct 10, 2026

Uh oh!

tensorrt-cicd commented Oct 10, 2026

Uh oh!

tensorrt-cicd commented Oct 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chenfeiz0326 commented Oct 8, 2026 •

edited

Loading

coderabbitai Bot commented Oct 8, 2026 •

edited

Loading