Repository navigation
fix(agentx): export AIPERF_VERSION so the aiperf mmap cache key tracks the pinned revision / 导出 AIPERF_VERSION 使 aiperf mmap 缓存键跟随固定的 aiperf 版本 - #3729
Closed
redhairdragon wants to merge 2 commits into
Conversation
…s the pinned revision / 导出 AIPERF_VERSION 使 aiperf mmap 缓存键跟随固定的 aiperf 版本 aiperf namespaces its memory-mapped dataset cache key with AIPERF_VERSION (utils/aiperf/src/aiperf/dataset/mmap_cache.py, _aiperf_version). InferenceX never set it, so the key carried no source-code component and a loader change in utils/aiperf that did not bump MANIFEST_VERSION kept replaying the previously decoded dataset from a warm /aiperf_mmap_cache. agentx-harness deb5421c (idle-gap cap moved from a load-time timeline warp to the runtime scheduler) was such a change: a cluster whose cache predates it replays the warped timeline (6.31 GiB dataset.dat, key e33b683c...) while a fresh cluster builds the unwarped one (6.30 GiB) under the same key, with 10-23% lower mean ISL and 5-23% lower input throughput per GPU on the same hardware and engine across four Kimi-K3 AgentX configs. Resolve the pinned utils/aiperf commit without git (gitdir pointer file, loose or packed ref), fall back to git rev-parse, then to a content hash of src/aiperf/dataset when the gitdir is not mounted in the container, and export it from install_agentic_deps before the aiperf CLI runs. A preset AIPERF_VERSION is left untouched. Companion to the agentx-harness MANIFEST_VERSION 26 -> 27 bump.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
aiperf namespaces its memory-mapped dataset cache key with
AIPERF_VERSION(utils/aiperf/src/aiperf/dataset/mmap_cache.py,_aiperf_version()→os.environ.get("AIPERF_VERSION") or None). InferenceX never sets it, so the key carries no source-code component and anyutils/aiperfchange that alters the decoded dataset without bumpingMANIFEST_VERSIONkeeps replaying the previously decoded dataset from a warm/aiperf_mmap_cache.That happened with agentx-harness
deb5421c(idle-gap cap moved from a load-time timeline warp to the runtime scheduler,MANIFEST_VERSIONleft at 26): the persistent cache ongb200-nvpredates it and still serves the warped timeline under keye33b683c648e0cd197606a97c1e2a9f0(dataset.dat6.31 GiB), while a fresh cluster builds the unwarped dataset under the same key (6.30 GiB). Becausetrajectory_source.pysamples the starting turn against the cached timestamps, the two caches pick differentroot_next/livebranches for identical(seed, lane, trace, sample_time).Measured on the same 16-GPU GB200 nodes, image and engine flags, aiperf CLI at the pinned
754356e9in both arms, only the cache directory differing (fresh build vs a rebuild of the same key withaiperf @ be758d62, which reproduces thegb200-nvlane/credit signature line for line):dcp16-dspark4 c16(442689)754356e9be758d62rebuildtp8pp2 c16(442692)754356e9be758d62rebuildtp8pp2 c72(442688)754356e9be758d62rebuildtp8pp2 c96(442695)754356e9be758d62rebuildSame hardware, fresh vs
be758d62arm: mean ISL −10…−23 %, prefix-cache hit −0.19…−3.37 pp, In Tput/GPU −5.0…−22.6 %. Warmup credits/records differ identically on every run (c16 17/177 vs 20/180; c72 76/796 vs 97/817; c96 101/1,061 vs 122/1,082; Official = thebe758d62values).Change (
inferencex-e2e/benchmarks/benchmark_lib.sh):export_aiperf_version()resolves the pinnedutils/aiperfcommit without git (the submodule's.gitis a gitdir pointer file; rootless Enroot containers may not ship git): readsHEAD, follows a symbolic ref to a loose or packed ref, then falls back togit rev-parse, and finally to a content hash ofsrc/aiperf/dataset/**/*.py(src-<12 hex>) when the gitdir is not mounted in the container. A presetAIPERF_VERSIONis left untouched; if nothing resolves it warns and leaves the variable unset (today's behaviour).install_agentic_deps()calls it before its early return, so the export is in place for every path that later runs$AIPERF_CLI profile(srt_agentic.sh,ensure_hf_cliinside the replay helper).utils/aiperfbump now gets its own cache namespace (one rebuild, ≈6.3 GiB for the Kimi-K3 corpus), and a harness change can no longer be masked by a warm cache.Companion fix upstream: SemiAnalysisAI/agentx-harness PR bumping
MANIFEST_VERSION26 → 27 (SemiAnalysisAI/agentx-harness#48). When that lands, the nextutils/aiperfbump invalidates pre-deb5421ccaches even without this export; this PR makes the invalidation independent of anyone remembering to bump the manifest.Verification:
bash -n; the helper was exercised standalone against (1) the real submodule checkout (detached HEAD via gitdir pointer → matchesgit rev-parse --short=12 HEAD), (2) a symbolic HEAD with a loose ref, (3) packed-refs only, (4) a relative gitdir pointer, (5) no.git→ stablesrc-content hash that changes when a dataset source changes, (6) presetAIPERF_VERSIONhonoured, (7) nothing resolvable → warning, unset, rc 0, (8) value visible to child processes — 13/13 checks pass.pytest -c inferencex-e2e/pyproject.toml inferencex-e2e/infx/tests/ -n 4(uv-syncedtestgroup, Python 3.12): the five modules that sourcebenchmark_lib.sh(results/agentic/test_power_lifecycle.py,results/power/test_process_result.py,evals/test_run_eval_dispatch.py,evals/test_batched_eval.py,workflows/test_launch_layout.py) → 239 passed; the fullinfx/testsrun gives the same 2,031 passed / 34 failed / 6 errors on this branch and onorigin/mainin the same environment (failures are local-only:utils/srt-slurmsubmodule not initialised in the worktree →srtctlimport /Missing srt-slurm submodule,uv sync --lockedpublish-time resolution, GitPythonrefs/heads/.invalid).Perf-changelog: not appended. This PR changes no recipe, engine or image; on clusters whose
/aiperf_mmap_cacheis already a post-deb5421cbuild the only effect is one cache rebuild. Ongb200-nv(stale pre-deb5421ccache) the AgentX replay moves to the unwarped timeline thatdeb5421calready introduced upstream in July, so the next AgentX sweeps there will show the ISL / hit / In-Tput shift above. If maintainers prefer those curves re-generated explicitly, say so and I will append a changelog entry listing the affectedkimik3-*gb200*agentic*config keys.AI model disclosure
Related Issue
n/a (companion: SemiAnalysisAI/agentx-harness#48)
Type of Change
Checklist
inferencex-e2e/perf-changelog.yamland have not edited historical entries (not appended — no recipe/engine change; see "Perf-changelog" above)/use <run_id>on this PR中文
改动说明： aiperf 用
AIPERF_VERSION给 mmap 数据集缓存键加命名空间（mmap_cache.py的_aiperf_version()），InferenceX 从未设置它，所以缓存键里没有任何源码成分。agentx-harnessdeb5421c把 idle-gap 上限从"加载时压缩时间线"改成"运行时调度"，但没有 bumpMANIFEST_VERSION（仍为 26），导致gb200-nv上早于该提交生成的缓存一直以同一个键e33b683c…回放旧的（压缩过的）时间线（dataset.dat6.31 GiB），而新集群生成的是未压缩版本（6.30 GiB）。trajectory_source.py按缓存里的时间戳采样起始轮次，两种缓存会选出不同的root_next/live分支。在同一批 GB200 节点、同一镜像与引擎参数、同一 aiperf CLI（
754356e9）下只更换缓存目录做 A/B（见上表）：平均 ISL 低 10–23%，前缀缓存命中率低 0.19–3.37 pp，每 GPU 输入吞吐低 5.0–22.6%；warmup credits/records 在每次运行中都以相同方式分叉（Official 等于be758d62一侧的值）。本 PR： 在
benchmark_lib.sh新增export_aiperf_version()，不依赖 git 解析utils/aiperf固定的 commit（gitdir 指针文件 → HEAD → loose/packed ref →git rev-parse→ 最后回退到src/aiperf/dataset源码内容哈希），在install_agentic_deps()的早退之前导出；已设置的AIPERF_VERSION不覆盖；无法解析时只打印 WARNING 并保持现状。之后每次 bumputils/aiperf都会得到独立的缓存命名空间（一次性重建约 6.3 GiB）。AI 模型使用说明： 使用 Google Antigravity（IDE 内的 agentic 编码助手）完成根因分析、A/B 实验设计与日志分析、shell 改动与独立测试、以及本说明；运行环境未暴露底层模型的确切标识，无法核实。另一 Antigravity agent（同一运行环境，同样无法核实标识）独立从原始日志复核了上表数据。最终由人工审阅。
关联 issue： 无；配套上游 PR：https://github.com/SemiAnalysisAI/agentx-harness/pull/48。
改动类型： Bug 修复。
验证：
bash -n；辅助函数单独测试 8 类场景共 13 项检查全部通过（真实 submodule、loose ref、packed-refs、相对 gitdir 指针、无.git时的内容哈希稳定且随源码变化、预设值保留、不可解析时告警且 rc=0、子进程可见）。注意事项： 未追加 perf-changelog 条目——本 PR 不改任何 recipe / 引擎 / 镜像；对已是新缓存的集群只会触发一次重建；对
gb200-nv这类仍是旧缓存的集群，AgentX 回放会切换到deb5421c七月就已引入的未压缩时间线，后续 sweep 会出现上表所示的 ISL / 命中率 / 输入吞吐变化。如维护者希望显式重新生成这些曲线，我可以补一条列出受影响kimik3-*gb200*agentic*config keys 的 changelog 条目。