[None][refactor] Unify performance collection and exposure

docs/source/commands/trtllm-serve/index.rst

@@ -7,4 +7,5 @@ trtllm-serve
 
    trtllm-serve
    render-endpoints
+   performance-metrics-collection
    run-benchmark-with-trtllm-serve

docs/source/commands/trtllm-serve/performance-metrics-collection.md

@@ -0,0 +1,129 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Shared performance metrics collection
+
+## Shared collection and exposure
+
+Performance measurements are collected before selecting their output. The
+backend-independent `PerfMetricsCollector` creates owned immutable
+`PerfMetricsRecord` values with three categories:
+
+| Category | Measurements |
+| --- | --- |
+| `per_req` | Candidate-completion latency/tokens/cache metrics, or explicitly enabled diagnostic request iteration records. Diagnostic request records are not a second completion stream. |
+| `per_iter` | Completed-batch iteration data and separately identified loop-boundary observations: host/device timing, scheduled/active/unscheduled/dummy work, fetched requests, and model state. |
+| `per_rank` | Rank-owned KV snapshots, memory and capacity limits. Actual snapshot capture iteration/time remain explicit; `batchIter` separately identifies the batch whose processing triggered collection. |
+
+Request and iteration records also retain their source rank. A delayed batch
+record and its attached `executorLoop` can describe different iterations under
+overlap or pipeline parallelism. The loop has its own iteration, rank and
+`loopSequence`; the sequence advances even while the executor iteration counter
+is stalled. Free-form model state and iteration identifiers remain diagnostic
+data, not Prometheus labels.
+
+`LoggerExposure` formats collected loop records in the executor thread using
+the existing iteration-line fields. It also writes structured `per_rank`
+records when a new rank snapshot is collected. With `enable_iter_req_stats`
+and `enable_iter_perf_stats`, request iteration diagnostics are available to
+the logger as categorized records. Detailed rank snapshots are exposed when
+collected, so the final snapshot need not wait for another executor loop.
+
+`PrometheusExposure` is the long-lived serving-side adapter. HTTP and OpenEngine
+normalize transported records through the same category model before exposing
+them. `MetricsCollector`, `log_iteration_stats` and `log_request_metrics_dict`
+remain compatibility interfaces. Scraping and idle periods do not trigger
+source reads, reset counters or require an active request.
+
+Neither exposure reads an executor/KV manager or drains native events. Source
+collection alone owns those operations. When batch stats are enabled, the loop
+logger reuses collected rank snapshots without an additional KV query. In
+logger-only mode the collector retains the existing basic-KV refresh cadence
+of 16 loop boundaries on selected logging ranks. Detailed KV/tier collection
+still requires `enable_iter_perf_stats`; enabling text alone does not enable
+all expensive request/tier diagnostics.
+
+| Use case | Existing controls |
+| --- | --- |
+| Production Prometheus | `return_perf_metrics: true`, `enable_iter_perf_stats: true`, `print_iter_log: false` |
+| Offline iteration text with basic KV utilization | `print_iter_log: true`; no HTTP server or Prometheus client instance is required |
+| Offline text plus detailed rank observations | `print_iter_log: true`, `enable_iter_perf_stats: true`; optionally enable request iteration diagnostics |
+| Both outputs | Enable iteration text alongside production collection; source reads and instrumentation are shared |
+
+This separation adds record construction/serialization and optional logging
+work. It introduces no CUDA events or synchronization beyond the existing
+collection/timing settings. The existing device-step profiler timing policy is
+preserved. CPU and
+end-to-end overhead of this refactor have not been benchmarked.
+
+## Transport cost and KV utilization semantics
+
+Worker serialization parses native JSON into a dictionary, partitions it into
+owned immutable categories, and encodes the existing JSON transport shape. The
+consumer still parses that JSON. Parsing the native JSON and encoding/decoding the
+wire payload were already required; partitioning, recursive freezing and mapping
+adaptation add CPU work. The worker's freeze runs on the transport thread. Source
+snapshot freezing and loop-record construction also run on the executor thread.
+
+The pending overhead benchmark must cover both source collection and transport:
+compare baseline and refactored CPU time/allocations per record and transport
+throughput with request diagnostics disabled/enabled, varying request counts,
+KV snapshot sizes and rank counts. Separately compare logger-only, Prometheus-only
+and combined outputs in an inference workload. No measured overhead claim is
+made before those checks run.
+
+KV utilization prefers an explicit `usedNumBlocks` value, including measured zero.
+For basic snapshots without that field, it uses `maxNumBlocks - freeNumBlocks`.
+Missing or non-positive capacity produces unavailable utilization (`N/A` in the
+iteration log). This preserves the logger-only free-block calculation and its
+zero-capacity `N/A` behavior. Prometheus retains its existing requirement for
+explicit used blocks and positive capacity; it does not infer a utilization series
+from a free-only snapshot. Reclaimable cached prefixes can remain in available
+blocks, so utilization alone does not measure conversation-cache retention.
+
+## Create a Grafana dashboard
+
+The [Grafana setup example](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/serve/grafana)
+provides a standard-library Python renderer and publisher for Kubernetes serving.
+It discovers engine workers and has E2E, Prefill, Decode and All views. Configure
+your workload pod/job patterns, datasource UIDs, endpoint/container labels and
+Grafana's supported schema-v2 API version in a copy of `example.config.json`.
+The default configuration generates metrics only; Loki, scrape-resource creation
+and trace-PVC panels are optional.
+
+```sh
+cp examples/serve/grafana/example.config.json /tmp/trtllm-dashboard.json
+# Edit the copied configuration for your deployment.
+python3 examples/serve/grafana/dashboard.py render \
+  --config /tmp/trtllm-dashboard.json --output /tmp/trtllm-dashboard-rendered
+python3 examples/serve/grafana/dashboard.py deploy \
+  --config /tmp/trtllm-dashboard.json --output /tmp/trtllm-dashboard-rendered --dry-run
+```
+
+Rendering is offline. The deployment dry-run validates the selected Grafana API
+and enabled Kubernetes resources without publishing them. Run `deploy` without
+`--dry-run` to publish. Use API persistence for API-owned dashboards and
+`configmap-v2` for existing file-sidecar ownership; applying its generated ConfigMap
+alone is incomplete. The tool guards resource versions and reads back saved
+content. ConfigMap checkpoint plus native finalization is not a cross-system
+transaction; the README documents `--finalize-only` recovery.
+
+The 80 default metric templates include physical tier states/resident occupancy,
+restore/eviction turnover, final drops, sampled restore latency/skips and
+completed-request transfer duration. Native ratios use logical byte capacity and
+remain separated by level/tier and reporting rank. Unsupported observations hide
+optional panels. GPU active/available block charts retain their separate meaning:
+available blocks can contain reclaimable prefixes. These panels cannot prove
+which conversation prefix was evicted.
+
+Select one application scrape job path to avoid duplicate engine observations.
+Existing `irate` charts use the last two samples in `counter_lookback`; new tier
+rate/turnover charts average the window, and sampled latency uses a 5-minute
+histogram window. Missing metrics, sparse timing samples, rank coverage and native
+build requirements still apply; dashboard creation does not enable instrumentation.
+
+The template includes optional KV-cache tier and transfer panels whose added
+instrumentation is provided separately by [PR #22083](https://github.com/NVIDIA/TensorRT-LLM/pull/22083).
+This metrics refactor does not introduce those metric families.

examples/serve/README.md

@@ -1,3 +1,16 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
 # Online Serving Examples with `trtllm-serve`
 
 We provide a CLI command, `trtllm-serve`, to launch a FastAPI server compatible with OpenAI APIs, here are some client examples to query the server, you can check the source code here or refer to the [command documentation](https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve.html) and [examples](https://nvidia.github.io/TensorRT-LLM/examples/trtllm_serve_examples.html) for detailed information and usage guidelines.
+
+
+## Grafana monitoring
+
+The [Grafana setup example](grafana/README.md) renders and publishes role-aware
+Kubernetes inference dashboards, including KV-cache pressure, transfer and
+executor performance views. Start with an offline render and the neutral example
+configuration; Grafana/Prometheus deployment and inference settings remain separate.

examples/serve/grafana/README.md

@@ -0,0 +1,178 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Grafana dashboards for Kubernetes serving
+
+Render and publish role-aware TensorRT-LLM monitoring dashboards using Python 3.10+
+and its standard library. Copy this whole directory, including `templates/`, to
+use it separately from the TensorRT-LLM checkout. Rendering does not import
+TensorRT-LLM, resolve credentials, contact Grafana, or invoke Kubernetes.
+
+The metrics dashboard has E2E, Prefill, Decode and All views. Engine workers are
+discovered from current scrape targets on refresh. The 80 default panel templates
+include request/queue latency, throughput, rank-local KV reuse/load, transfer
+outcomes, executor timing and physical tier pressure. Optional trace-PVC charts
+add two templates. Optional Loki support adds a separate 16-panel Logs dashboard.
+Auto-grid panels without observations are hidden; missing native tier/timing
+measurements do not become zero-valued metrics.
+
+## Requirements
+
+- A Grafana server exposing the **dashboard schema v2** API and dynamic layouts.
+  `grafana.api_version` selects `v2` (the example) or `v2beta1`. This is not the
+  Classic dashboard JSON API. Confirm the selected version in your server's
+  Swagger and run the validation command before publication. The `12.3.1` plugin
+  versions in templates do not establish server API compatibility: Grafana's
+  [v12.3.1 schema snapshots](https://github.com/grafana/grafana/tree/v12.3.1/pkg/tests/apis/openapi_snapshots)
+  expose `v2alpha1`, which this example does not target. The current
+  [Grafana dashboard API documentation](https://grafana.com/docs/grafana/latest/developer-resources/api-reference/http-api/dashboard/)
+  recommends `v2`.
+- A Prometheus datasource and application series with `namespace`, `pod`, `job`
+  and `endpoint` scrape labels. Set `metric_job_regex` to **one application scrape
+  path**, especially when both frontend and engine endpoints expose overlapping
+  counters. Selectors are deployment inputs, not inferred from the current cluster.
+- kube-state-metrics for deployment topology; DCGM with `exported_namespace` and
+  `exported_pod` labels for GPU charts. Dynamo frontend metrics enable the E2E
+  charts. Missing optional exporters do not establish workload health.
+- `kubectl` is needed for ConfigMap publication, Kubernetes Secret authentication,
+  or optional Service/ServiceMonitor creation. API publication with environment
+  authentication and scrape setup disabled makes no Kubernetes calls.
+- Loki is optional and disabled in the example. Its charts expect the Dynamo/TRTLLM
+  iteration/trace log fields represented in `templates/logs.json`; this tool does
+  not install a log collector. Optional distinct-session queries count observed
+  trace IDs, not online connections, and can encounter Loki cardinality limits.
+
+## Configure and render
+
+From the repository root:
+
+```sh
+cp examples/serve/grafana/example.config.json /tmp/trtllm-dashboard.json
+# Edit /tmp/trtllm-dashboard.json for your deployment and datasource UIDs.
+python3 examples/serve/grafana/dashboard.py render \
+  --config /tmp/trtllm-dashboard.json --output /tmp/trtllm-dashboard-rendered
+```
+
+Rendering produces `metrics.json`, optionally `logs.json`,
+`dashboards.configmap.json`, and `scrape-resources.json`. These are local artifacts;
+rendering alone does not create a dashboard. Keep deployment configs and rendered
+resource identities outside the distributable example.
+
+| Configuration | Meaning |
+| --- | --- |
+| `kubernetes.context`, `namespace` | Explicit cluster context and workload namespace; no current-context fallback. |
+| `grafana.url`, `api_namespace`, `api_version` | Grafana endpoint, API namespace and supported schema-v2 API version. |
+| `grafana.namespace`, `configmap`, `sidecar_labels` | Monitoring namespace and existing sidecar configuration for file publication. |
+| `grafana.publication` | `api` or `configmap-v2`, according to who owns the dashboard. |
+| `grafana.dashboard_keys` | Existing metrics/logs JSON filenames in the ConfigMap; preserve them when updating a file-owned dashboard. |
+| `dashboard.*_uid`, `title`, `refresh` | Stable dashboard identities, display title and refresh period. |
+| `dashboard.include_logs` | Enable the optional Logs dashboard; defaults to false in this example. |
+| `datasources.prometheus`, `loki` | Grafana datasource UIDs; the Loki value is unused when Logs is disabled. |
+| `workload.*_pod_regex`, `metric_job_regex` | Workload selection and the application scrape path. |
+| `workload.metric_role_label`, `decode_metric_role_value` | Exporter role label and decode value; ServiceMonitors can inject this label. |
+| `workload.engine_endpoint`, `engine_container`, `frontend_container` | Actual endpoint and container labels. |
+| `workload.log_*`, `frontend_container` | Loki job, role and container contract, when Logs is enabled. |
+| `workload.trace_pvc` | Optional trace volume; null omits both PVC charts. |
+| `scrape.enabled` | Generate/apply scoped Services and ServiceMonitors; false by default. |
+
+Use the supported RE2-compatible subset: ordinary groups and `[.]` for literal
+periods; no lookaround, inline flags, backslashes, quotes, colons or commas. Use
+JSON booleans, not strings such as `"false"`.
+
+`dashboard.counter_lookback` defaults to `1m`; choose at least two actual scrape
+intervals. Existing `irate` panels use the last two scrapes within that lookup
+window. New tier turnover/drop panels use `rate`, averaging resets-aware deltas
+across the window. Migration/transfer latency percentiles use a fixed 5-minute
+bucket-rate window. Refresh cadence, scrape cadence and statistics collection
+cadence are independent.
+
+## Authentication and publication
+
+`grafana.auth` accepts references, not inline credentials:
+
+```json
+{"token_env": "GRAFANA_TOKEN"}
+```
+
+Or use `username_env` / `password_env`, or an existing Kubernetes Secret reference:
+
+```json
+{"secret": "grafana-auth", "username_key": "admin-user", "password_key": "admin-password"}
+```
+
+Supply values through your environment or secret manager. Use HTTPS or a loopback
+administrative tunnel. The client rejects redirects and omits credentials, raw
+HTTP error bodies and Kubernetes stderr from output.
+
+Validate the configured server/API and any enabled Kubernetes resources:
+
+```sh
+python3 examples/serve/grafana/dashboard.py deploy \
+  --config /tmp/trtllm-dashboard.json --output /tmp/trtllm-dashboard-rendered --dry-run
+```
+
+This uses server-side `dryRun=All` / Kubernetes dry-run and writes local artifacts;
+it does not publish dashboards or scrape resources. Publication is explicit:
+
+```sh
+python3 examples/serve/grafana/dashboard.py deploy \
+  --config /tmp/trtllm-dashboard.json --output /tmp/trtllm-dashboard-rendered
+```
+
+`api` saves directly in Grafana. Use `configmap-v2` when an existing file
+sidecar owns these UIDs; an API-only update can be overwritten by that provisioner.
+The latter mode preserves unrelated ConfigMap entries/metadata, applies the
+checkpoint, waits for its source checksum, then finalizes and verifies the native
+v2 spec. Applying the rendered ConfigMap alone does not complete this workflow.
+
+The script checks ConfigMap data/resource versions and dashboard resource versions
+before writes, including after a sidecar checkpoint. Final native PUTs retain
+`resourceVersion` for optimistic concurrency. This is not a transaction spanning
+Kubernetes and Grafana: partial checkpoints can exist after an error. Resume only
+with the same config/rendered checkpoint:
+
+```sh
+python3 examples/serve/grafana/dashboard.py deploy \
+  --config /tmp/trtllm-dashboard.json --output /tmp/trtllm-dashboard-rendered --finalize-only
+```
+
+A successful publication reads back layouts, queries, variables, links, title and
+time settings and writes `receipt.json` with `verified: true`. Each invocation
+removes an old receipt first. Always check the current command's exit status.
+Publication changes only the configured dashboard/scrape resources; the example
+neither installs Grafana/Prometheus nor changes inference configuration.
+
+## Interpreting KV-cache panels
+
+Resident occupancy includes retained evictable prefixes. Combine it with turnover,
+final capacity/priority drops and recent misses; high occupancy alone does not
+prove thrashing. Capacity ratios weight pool groups by logical slot bytes. Legacy
+GPU active/available charts are not physical tier occupancy, and available blocks
+can contain useful prefixes. No host-only occupancy is inferred from aggregated
+secondary-tier statistics.
+
+Restore/evict turnover is `rate(moved_bytes) / capacity` in turnovers per second;
+it can exceed one per second. Capacity/priority drops measure final prefix loss,
+not ordinary request release. Native counters count accepted submissions and do
+not confirm successful asynchronous DMA. Detailed tier rank coverage can be limited.
+
+Sampled migration latency is disabled unless
+`TRTLLM_KV_CACHE_TIMING_SAMPLE_INTERVAL` is positive before manager construction.
+It measures batch stream intervals, including codec/staging, and is distinct from
+completed-request KV-transfer duration. Neither is necessarily a request's restore
+stall. Compare source/destination paths and skipped timing samples; sparse or
+absent samples do not mean zero latency. Ratios and latencies are not stacked.
+
+## Standalone tests
+
+From the repository root, the utility tests need Python, `pytest` for the CI marker,
+and no TensorRT-LLM runtime import. They mock publication; they do not deploy:
+
+```sh
+python3 -m unittest discover -s tests/unittest/metrics -p test_grafana_dashboard.py -v
+```
+
+A successful offline render or mocked publisher test is not a live Grafana,
+Prometheus-query or native KV-instrumentation integration test.