feat(server): publish cumulative prefill and decode timing on /v1/stats

[mgeldi]

Closes #552

What

Adds four cumulative counters to the requests object of /v1/stats, so a process polling the server can diff two polls into prefill and decode tokens/s (the same way llama.cpp's prompt_seconds_total / tokens_predicted_seconds_total are used):

Field	Meaning
prefill_seconds_total	per request: admission (new_user) to its first output reply
decode_seconds_total	per request: time between its later output replies
decode_tokens_total	tokens carried by those later replies
cached_prompt_tokens_total	summed cached_tokens (prefix-cache hits)

Everything is measured in StatsTracker, which already observes every reply in FrontendManager.listen(), so streaming and buffered requests are timed the same way and nothing depends on reasoning/tool parsers holding text back. Tokens that arrive on a request's first output reply (overlap can deliver several) are not counted as decode work, because no decode interval measured them. Per-request timing state is released on the terminal reply. Existing fields and the window rates are unchanged.

Tests

• New tests/server/test_stats_timing.py (5 tests): prefill/decode split, tokens on the first reply, two interleaved requests summed per request with state released, a request failing before output adds no time, and the fields in build_stats.
• pytest tests/server tests/scheduler/test_cost_accounting_core.py -m "not slow": 578 passed.
• ruff check --select E4,E7,E9,F on the changed files: clean.

Tested on

RTX 5090 32 GB, driver 615.71.09, CUDA 13.4, Ryzen 7 9800X3D, 91 GiB RAM, Linux 7.2.6 (CachyOS). Checkpoint nvidia/Qwen3.8-Flash-Next-NVFP4 (rev fc694b54) converted to FTW (b12x) with FreeToken, branch based on f5b9700:

ft serve --model <Qwen3.8-Flash-Next-NVFP4 FTW> --served-model-name hermes-brain \
  --max-seq-len-override 262144 --num-tokens 262144 --gpu 0 --max-running-requests 1 \
  --cuda-graph-max-bs 1 --max-prefill-length 16384 --memory-ratio 0.85 \
  --moe-strategy offload --moe-cache-size 2048 --ple-backend disk \
  --quant-backend moe.nvfp4=b12x --enable-cache-report \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder

Counters diffed around each request, compared with client-side timing from the SSE stream:

Request	/v1/stats delta	Client side
streamed, 17,804-token prompt, 300 out	prefill 9.82 s (ttft_mean_ms 9811), decode 299 tok / 7.53 s = 39.7 tok/s	TTFT 10.19 s, decode 7.53 s
buffered, 67-token prompt, 52 out	prefill 2.04 s, decode 51 tok / 1.29 s	total 3.34 s
same 17,804-token prompt again	cached_prompt_tokens_total +17,792	TTFT 1.94 s
streamed, 132-token prompt, 2048 out, x2	decode 35.3 and 36.0 tok/s	35.3 and 36.0 tok/s

The buffered request is the case the request ring cannot time today (ttft_ms is None there).

AI-assisted: the change and tests were written with Claude Code, and the unit tests and live runs above were run on the hardware listed. I have reviewed the change.

🤖 Generated with Claude Code