[Performance]: Keep relayed prompt IDs on the int32 array path #21998

Proposal to improve performance

With gen_tokids_ctxbytes, a context worker returns the tokenized prompt as a
base64 int32 buffer and the orchestrator relays that string without building a
Python integer list. The generation worker currently ends that fast path with
np.frombuffer(...).tolist(). GenerationRequest then converts the list back
to int32 bytes for proxy submission, and its GenerationResult retains one
Python integer per prompt token until the request completes.

#19658 added a direct ndarray path to GenerationRequest and identified this
frontend handoff as follow-up work. The generation worker can pass the decoded
array in the token-prompt dictionary for both Chat Completions and Responses.
Explicit prompt_token_ids lists remain unchanged.

CPU benchmark

Intel Xeon Platinum 8352Y, CPython 3.12.3, one pinned CPU core. The benchmark
uses the repository's base64 decoder, GenerationRequest, and pickle submit
path with identical int32 inputs. It runs five warmups, then nine runs of ten
operations and reports the median per operation.

Prompt tokens	Current list path	Array path	Speedup	Python peak allocation
40,000	8.328 ms	0.692 ms	12.03x	1.90 -> 0.69 MiB
131,072	18.759 ms	1.553 ms	12.08x	6.25 -> 2.25 MiB
290,000	61.352 ms	9.384 ms	6.54x	13.82 -> 4.98 MiB

Pickle round trips produce identical prompt-token lists. The removed work is
on the generation worker's per-request TTFT path, and the list otherwise stays
resident for the request lifetime.

Current environment

• OS: Linux x86_64

• Python: 3.12.3

• CPU: Intel Xeon Platinum 8352Y

• GPU: not used

• Base: current main at 219259709

• I searched open and closed issues and pull requests, including recent
  prompt_token_ids_b64, array-prompt, performance, memory, and disaggregated
  serving work.

AI assistance (Cursor) was used to investigate and prepare this report.