[akyang-anyscale]

Description

Reverts #66325 so we can run the Serve microbenchmark release test against it. This is a draft for performance testing and is not meant to be merged as is.

In the nightly serve_throughput_optimized_microbenchmarks release test, handle_model_comp_avg_rps dropped from about 1,017 to about 143 requests per second between build 108436 (commit be427bacd2, 2026-09-25) and build 108516 (commit 5b14d650d7, 2026-09-26). Of the 15 commits in that window, #66325 is the only one that changes the Serve request path. The other candidate, #65660, makes a small change to lineage reconstruction in reference_counter.cc.

A possible mechanism, which is not yet verified: after #66325, a routing task is cancelled when another task fulfills its request. As #66632 describes, the queue-length probes that task was waiting on are not cancelled with it, so their get_num_ongoing_requests calls keep running against replicas. If the throughput recovers on this branch, #66632 is a good candidate to test next as a forward fix.

Caveat: this revert reintroduces the bug #66325 fixed. Under routers that match assignments to specific request IDs, a routing task can retire before it assigns its request, which can leave that request waiting indefinitely.

Related issues

• Reverts [serve] Prevent routing tasks from abandoning pending requests #66325
• Related: [serve] Deflake test_pow_2_request_router - Cancel queue length probes when their routing task is cancelled #66632 (forward fix for routing task cancellation leaving queue-length probes running)

Duplicate check

I searched open PRs referencing #66325. The only one is #66632, which is a forward fix for a flaky test, not a revert.

Tests

• No tests were run locally. The development machine had no Python environment with Ray installed, and the pre-commit hooks failed to install because they require Python 3.9 or newer.
• The revert applied cleanly on current master. The later change to request_router.py from [serve][LLM] Call on_request_routed for pick-only replica selections #66489 is untouched. The revert deletes python/ray/serve/tests/unit/test_request_router.py, which [serve] Prevent routing tasks from abandoning pending requests #66325 added. The unit test Bazel target collects tests with glob(["test_*.py"]), so no target references it by name.
• Planned: run the Serve microbenchmark release test against this branch and compare handle_model_comp_avg_rps with master.

AI assistance

AI assistance (Claude Code) was used to identify the suspect commit, prepare the revert, and write this description.

🤖 Generated with Claude Code