[hyukn]

Description

Two related fixes to AutoTunerProfilingCache (tensorrt_llm/_torch/autotuner.py) so concurrent jobs sharing TLLM_AUTOTUNER_CACHE_PATH stop losing work and the cache file is no longer corrupted by a writer killed mid-write.

1. Merge, do not replace, the per-rank dict on save

current_cache[f"rank_{rank}"] = serialized_rank_cache was wiping every other concurrent writer's rank_{rank} contribution and any preexisting on-disk entries the current process did not load.

Replaced with the same setdefault(...).update(...) pattern already used for the shared dict.

Verified by a 4-job synthetic stress (cache-race-multi-job/): 0 of 3 lost-update rounds after fix vs 3 of 3 before.

2. Move locking off the data file and replace truncate+json.dump with write-tmp + fsync + os.rename

The previous code held fcntl.flock(LOCK_EX) on the data file in 'a+' mode and did seek + read + truncate + json.dump in-place. Two real failure modes:

• A writer killed mid-json.dump (preempt / OOM / scancel / node failure) leaves a partially-written file and every subsequent reader fails with JSONDecodeError. Observed on a 60-job stress with a payload ~30 MB.
• O_APPEND from 'a+' plus truncate semantics are filesystem-quirky and make the failure unrecoverable across nodes.

New design:

• fcntl.lockf(LOCK_EX) on a separate persistent lockfile next to the data file (POSIX byte-range lock; more portable across Lustre / NFS than BSD flock(2)). Lockfile is never unlinked; the kernel releases the lock on fd close including process death, so no stale-lock cleanup path is needed.
• The data file body is written to <name>.tmp.<pid>.<ns>, fsync'd, and then os.rename'd. A killed writer leaves only a discardable tmp file; the data file is never partially written, so concurrent readers always observe a fully-formed JSON document.

Relation to PR #14458

The TuningConfig.exclude_from_cache flag and single-(runner, tactic) profile shortcut for JIT-driver ops were originally bundled here. They are now split into PR #14458, opened independently against feat/deepseek_v4. The two PRs do not depend on each other.

Test Coverage

Validated end-to-end on lyris with the following stress harnesses (in this repo under cache-race-multi-job/ and cache-race-e2e-v3lite/):

• Synthetic per-writer race repro — 4 SLURM jobs × 4 procs each, barrier-synchronised + 10 s hold inside autotune() so save_cache calls all overlap. Each writer tunes a unique op-name + payload so lost entries are detectable by counting. Before fix: 3 of 3 rounds dropped 3 of 4 writers' contributions; after fix: 0 of 3 rounds drop any writer.
• Cross-node counter test — 60 SLURM jobs × 4 procs × 50 iters incrementing one shared file. Counter ends at 12 000 / 12 000 with the new lockfile-based code, confirming POSIX byte-range locks serialise correctly cross-node on this Lustre.
• Single-host RMW stress — 16 procs × 300 iters of save_cache pattern. With the previous 'a+' + flock design and the lock disabled, the file corrupts within the first 10 iterations (JSONDecodeError: Extra data ...). With the new design (separate lockfile + atomic rename) the file remains valid JSON across 9 600 RMW ops.
• E2E DS-V3-Lite — multi-job trtllm-bench throughput --tp 4 runs sharing one TLLM_AUTOTUNER_CACHE_PATH. Every writer's contribution survives in the on-disk file; no crashes.

No new unit tests added in this PR (the existing tests/unittest/_torch/misc/test_autotuner.py has zero coverage of save_cache / load_cache — adding multi-writer coverage is a worthwhile follow-up but out of scope for this fix).

PR Checklist

• PR description clearly explains what and why.
• PR Follows TRT-LLM CODING GUIDELINES to the best of my knowledge.
• No new dependencies introduced.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[hyukn]

/bot run --disable-fail-fast --add-multi-gpu-tests

[tensorrt-cicd]

PR_Github #49880 Bot args parsing error: usage: /bot [-h]
{run,kill,skip,submit,reviewers,reuse-pipeline,reuse-review} ...
/bot: error: unrecognized arguments: --add-multi-gpu-tests

Link to invocation

[hyukn]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #49887 [ run ] triggered by Bot. Commit: 1c510db Link to invocation

[tensorrt-cicd]

PR_Github #49887 [ run ] completed with state SUCCESS. Commit: 1c510db
/LLM/main/L0_MergeRequest_PR pipeline #39466 completed with status: 'SUCCESS'

CI Report

Link to invocation

[hyukn]

/bot run --disable-fail-fast --add-multi-gpu-test

[tensorrt-cicd]

PR_Github #50157 [ run ] triggered by Bot. Commit: 1c05254 Link to invocation

[tensorrt-cicd]

PR_Github #50157 [ run ] completed with state SUCCESS. Commit: 1c05254
/LLM/main/L0_MergeRequest_PR pipeline #39702 completed with status: 'SUCCESS'

CI Report

Link to invocation