Repository navigation
[Feature]: Solve the Perf regression test large variance #8391
Description
🚀 The feature, motivation and pitch
The test_perf.py regression test shows large variance when run in CI. I see that it runs on two different device types: H100 NVL and H100 PCIE. according to that the perf varies and also the clock frequencies. Need to ability to define threshold per device type if possible, otherwise need to use only one device type by applying a filter in the ci definitions (devops said that they use two device kinds due to machine shortage). see the attached excel chart analyzing the perf variance in correlation to the clock freqs implying difference in perf is due to two different freq levels due to two different device types being used.
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Activity
[#8391][fix] check perf by device subtype (#8428)
[NVIDIA#8391][fix] check perf by device subtype (NVIDIA#8428)
[NVIDIA#8391][fix] check perf by device subtype (NVIDIA#8428)
[NVIDIA#8391][fix] check perf by device subtype (NVIDIA#8428)
[NVIDIA#8391][fix] check perf by device subtype (NVIDIA#8428)
[NVIDIA#8391][fix] check perf by device subtype (NVIDIA#8428)
[#8391][chore] test_perf.py to lock clocks read from gpu_configs.yml …
[NVIDIA#8391][chore] test_perf.py to lock clocks read from gpu_config…
[TRTLLM-7963][feat] Cold L2 cache when doing autotune benchmarking. (N…
[NVIDIA#8391][chore] test_perf.py to lock clocks read from gpu_config…
[#8391][chore] added deepseek_r1_distill_qwen_32b AutoDeploy perf tes…
[#8391][chore] removed llama and added deepseek to AutoDeploy's L0 pe…
[NVIDIA#8391][chore] added deepseek_r1_distill_qwen_32b AutoDeploy pe…
[NVIDIA#8391][chore] removed llama and added deepseek to AutoDeploy's…