[redhairdragon]

Summary

deb5421c (fix(agentx): enforce per-trajectory runtime idle caps, 2026-07-30) moved --trace-idle-gap-cap-seconds enforcement from a load-time timeline warp in the weka loader (_IdleGapTimeWarp, removed from src/aiperf/dataset/loader/weka_trace.py) to the runtime scheduler. That changes Turn.timestamp / Turn.delay and the per-conversation start_timestamp_ms / end_timestamp_ms decoded for a given cache key, but MANIFEST_VERSION stayed at 26 (last bumped in cef80df4, 2026-07-24).

compute_cache_key() has no source-code component unless AIPERF_VERSION is set (_aiperf_version() returns None by default), so both be758d62 (pre-deb5421c) and every later revision produce the same key for the same CLI. A warm /aiperf_mmap_cache populated before deb5421c therefore logs Memory-mapped dataset cache HIT and keeps replaying the warped timeline — exactly the failure mode the contract comment at the top of mmap_cache.py warns about:

Bump when the on-disk layout, side-data schema, OR the decoded content the loaders produce for a given key changes -- the key has no source-code component, so a content-semantics fix must bump this or warm caches keep serving the old (wrong) dataset.

Evidence (InferenceX inferencex-agentx-mvp Kimi-K3 scenario, --random-seed 42 --num-dataset-entries 393 --trace-idle-gap-cap-seconds 300)

	pre-deb5421c build of the key (be758d62)	fresh build at 754356e9
cache key	e33b683c648e0cd197606a97c1e2a9f0	e33b683c648e0cd197606a97c1e2a9f0 (identical)
dataset.dat	6,770,325,731 B (logged 6.31 GiB)	6,769,044,663 B (logged 6.30 GiB)
manifest.json	57,464,090 B	56,182,450 B
lane 04 (02bc0afb, sample_time 47%)	root_next=0/13, live=13 (58 profiling requests, 287k–365k ISL)	root_next=0/13, live=1 (0 profiling requests)
lane 11 (069d7bf5, 49%)	root_next=149/219	root_next=210/219
warmup credits / records at conc 16 · 72 · 96	20 / 180 · 97 / 817 · 122 / 1,082	17 / 177 · 76 / 796 · 101 / 1,061

Because timing/trajectory_source.py samples t* = start + U(0.25, 0.75)·(end − start) against the cached timestamps, the two caches select different starting turns and live branches for identical (seed, lane, trace, sample_time). End to end on the same 16-GPU GB200 nodes, image and engine flags (aiperf CLI at 754356e9 in both arms, only the cache directory differs), the warped cache yields 10–23 % higher mean ISL, +0.19…+3.37 pp prefix-cache hit and 5–23 % higher input throughput per GPU across four Kimi-K3 AgentX configurations; the persistent cache on the InferenceX gb200-nv cluster (run 34836531846) matches the pre-deb5421c signature line for line.

Change

• MANIFEST_VERSION 26 → 27 with a # v27: note in the existing per-version comment block, so every pre-deb5421c entry misses and is rebuilt once on upgrade.
• tests/unit/dataset/test_mmap_cache.py: update the pinned version assertion and add test_lookup_rejects_pre_runtime_idle_cap_manifest (a version: 26 manifest must miss).

No key-payload or on-disk layout change.

Operational impact

One rebuild per cache directory (≈6.3 GiB / a few minutes for the Kimi-K3 corpus). Clusters that still had a pre-deb5421c cache will move to the unwarped timeline that deb5421c intended; their AgentX ISL / hit-rate / input-throughput numbers shift accordingly (see InferenceX companion PR SemiAnalysisAI/InferenceX#3729, which additionally exports AIPERF_VERSION so future harness bumps namespace the key automatically).

Testing

Local, Linux x86_64, CPython 3.12.15, uv pip install -e '.[test]' -e 'tests/aiperf_mock_server[dev]' + tools/generate_plugin_artifacts.py --enums/--overloads (as in run-unit-tests.yml):

• pytest tests/unit/dataset/test_mmap_cache.py → 72 passed (incl. the new test; the new test fails on master because a version: 26 manifest is still accepted there).
• ruff check / ruff format --check (0.14.14) and codespell clean on both touched files.

AI model disclosure

Prepared with Google Antigravity (agentic coding assistant). The runtime does not expose the exact underlying model identifier, so it could not be verified. Role: root-cause analysis of the cache-key behaviour, A/B experiment design and log analysis, patch and test drafting, and this description; an independent verification pass by a second Antigravity agent re-derived the numbers above from the raw logs. Final review by the human author.

---

Note

Low Risk
Version bump and unit tests only; forces cache misses and rebuilds without changing key computation or mmap layout.

Overview
Bumps the mmap dataset cache MANIFEST_VERSION from 26 to 27 so pre-upgrade warm entries are treated as a miss and rebuilt. The bump documents that --trace-idle-gap-cap-seconds is no longer applied via load-time timeline warping in the Weka loader; decoded Turn.timestamp / Turn.delay and per-conversation start/end timestamps for the same cache key differ, while the content-addressed key can stay identical when AIPERF_VERSION is unset.

Tests pin the new version and add test_lookup_rejects_pre_runtime_idle_cap_manifest, asserting manifests at version 26 no longer hit lookup. No on-disk layout or cache-key payload changes—only manifest gating and operational one-time rebuild per cache directory.

^{Reviewed by Cursor Bugbot for commit a50570e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@a50570e1c31b942c022ba7a689620ffc59573811

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@a50570e1c31b942c022ba7a689620ffc59573811

---

Last updated for commit: a50570e • Browse code