[AutoDeploy]: Pre-merge L0 cost-reduction

🚀 The feature, motivation and pitch

Move perf tests to post-merge. Currently perf/test_perf.py::test_perf[deepseek_r1_distill_qwen_32b-bench-_autodeploy-float16-kv_frac:0.8-input_output_len:1024,1024-reqs:512] is in pre-merge and accounts for a bug chunk of our budget. Consider if checkpoint is one of the crucial perf tests, or should we focus on models we care more about.
Reduce test variants, specifically in Nemotron tests. Focus on testing the model registry config in pre-merge, and move other variants to post-merge.

Full report link: https://gitlab-master.nvidia.com/junq/my-notes/-/blob/main/l0-pre-merge-4wk-2026-05-13/l0-pre-merge-duration-analysis.md

Alternatives

No response

Additional context

No response

Before submitting a new issue...

Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

@hnover-nv

Let's add this as a task as part of the broader epic to improve CI for AutoDeploy. @hnover-nv

AutoDeploy tests in `test_llm_api_autodeploy.py` — pre_merge vs post_merge

Scope: all entries under tests/integration/test_lists/test-db/l0_*.yml that reference accuracy/test_llm_api_autodeploy.py. AD tests only appear in: l0_l40s.yml, l0_h100.yml, l0_b200.yml, l0_dgx_h100.yml, l0_dgx_b200.yml. No other test-db YAML (gb200/gb300/h200/dgx_b300/etc.) references this file.

Totals

Stage	Entries
`pre_merge`	54
`post_merge`	11
Total	65

Note: TestNemotronUltraV3::test_accuracy[nvfp4-4] is listed in both the 4-GPU pre_merge and 4-GPU post_merge sections of l0_dgx_b200.yml — this is a duplicate.

Pre-merge

`l0_l40s.yml` — 1× L40S

TestNemotronNanoV3::test_accuracy[fp8-1-flashinfer]

`l0_h100.yml` — 1× H100 (backend=autodeploy)

TestLlama3_1_8B::test_auto_dtype[trtllm-False-1]
TestLlama3_1_8B::test_auto_dtype[trtllm-True-1]
TestLlama3_1_8B::test_auto_dtype[triton_paged-False-1]
TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[flashinfer]
TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[trtllm]
TestNemotronH::test_auto_dtype[trtllm-triton_ssm-False]
TestNemotronH::test_auto_dtype[trtllm-flashinfer_ssm-False]
TestNemotronH::test_auto_dtype[trtllm-triton_ssm-True]
TestGLM4Flash::test_auto_dtype[trtllm-False]
TestGLM4Flash::test_auto_dtype[trtllm-True]
TestNemotronV2::test_auto_dtype[False]
TestNemotronV2::test_auto_dtype[True]
TestNemotronV2::test_fp8[True]
TestNemotronNanoV3::test_accuracy[bf16-1-trtllm]
TestNemotronNanoV3::test_accuracy[fp8-1-trtllm]

`l0_b200.yml` — 1× B100/B200 (backend=autodeploy)

TestLlama3_1_8B::test_auto_dtype[trtllm-False-1]
TestLlama3_1_8B::test_auto_dtype[triton_paged-False-1]
TestLlama3_1_8B::test_auto_dtype[torch-True-1]
TestGLM4Flash::test_auto_dtype[trtllm-False]
TestGLM4Flash::test_nvfp4[False]
TestNemotronV2::test_auto_dtype[False]
TestNemotronV2::test_auto_dtype[True]
TestNemotronV2::test_fp8[True]
TestNemotronV2::test_nvfp4[True]
TestNemotronNanoV3::test_accuracy[fp8-1-trtllm]
TestNemotronNanoV3::test_accuracy[nvfp4-1-trtllm]
TestNemotronSuperV3::test_accuracy[nvfp4-1-attn_dp_off-trtllm]
TestNemotronSuperV3::test_functional_small[bf16]
TestNemotronSuperV3::test_functional_small[fp8]

`l0_dgx_h100.yml` — 4× H100 (backend=autodeploy)

TestLlama3_1_8B::test_auto_dtype[trtllm-False-4]
TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[flashinfer]
TestLlama3_1_8B_Instruct_Eagle3::test_eagle3_one_model[trtllm]
TestNemotronNanoV3::test_accuracy[bf16-4-trtllm]
TestNemotronNanoV3::test_accuracy[fp8-4-trtllm]
TestNemotronSuperV3::test_accuracy[bf16-4-attn_dp_off-trtllm]
TestNemotronSuperV3::test_accuracy[bf16-4-attn_dp_on-trtllm]
TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_off-trtllm]
TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_on-trtllm]
TestLlama3_1_8B::test_attention_dp[4]
TestQwen3_5_397B_MoE::test_bf16_small[4]
TestGemma4MoE::test_bf16
TestMiniMaxM2::test_finegrained_fp8

`l0_dgx_b200.yml` — 4× B200 (backend=autodeploy)

TestLlama3_1_8B::test_auto_dtype[trtllm-False-4]
TestNemotronNanoV3::test_accuracy[fp8-4-trtllm]
TestNemotronNanoV3::test_accuracy[nvfp4-4-trtllm]
TestNemotronSuperV3::test_accuracy[bf16-4-attn_dp_off-trtllm]
TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_off-trtllm]
TestNemotronSuperV3::test_accuracy[fp8-4-attn_dp_on-trtllm]
TestNemotronSuperV3::test_accuracy[nvfp4-4-attn_dp_on-trtllm]
TestNemotronSuperV3::test_mtp[ws4_180gb-flashinfer]
TestNemotronSuperV3::test_mtp[ws4_180gb-trtllm]
TestNemotronUltraV3::test_accuracy[nvfp4-4]
TestModelRegistryAccuracy::test_autodeploy_from_registry[nvidia_Llama-3.1-8B-Instruct-NVFP4-True]

Post-merge

`l0_dgx_h100.yml` — 4× H100 (backend=autodeploy)

TestModelRegistryAccuracy::test_autodeploy_from_registry[google_gemma-3-1b-it-False]
TestModelRegistryAccuracy::test_autodeploy_from_registry[meta-llama_Llama-3.1-8B-Instruct-False]
TestModelRegistryAccuracy::test_autodeploy_from_registry[mistralai_Ministral-8B-Instruct-2410-False]
TestModelRegistryAccuracy::test_autodeploy_from_registry[mistralai_Codestral-22B-v0.1-False]
TestModelRegistryAccuracy::test_autodeploy_from_registry[Qwen_QwQ-32B-False]
TestModelRegistryAccuracy::test_autodeploy_from_registry[meta-llama_Llama-3.3-70B-Instruct-False]
TestModelRegistryAccuracy::test_autodeploy_from_registry[nvidia_Llama-3.1-8B-Instruct-FP8-True]

`l0_dgx_b200.yml` — 4× B200 (backend=autodeploy)

TestNemotronUltraV3::test_accuracy[nvfp4-4] (also in pre_merge — duplicate)

`l0_dgx_b200.yml` — 8× B200 (backend=autodeploy)

TestModelRegistryAccuracy::test_autodeploy_from_registry[deepseek-ai_DeepSeek-R1-0528-True]
TestQwen3_5_397B_MoE::test_nvfp4[8]
TestNemotronUltraV3::test_accuracy[nvfp4-8]

Per-class breakdown

Test class	Pre-merge	Post-merge
`TestLlama3_1_8B`	7	0
`TestLlama3_1_8B_Instruct_Eagle3`	4	0
`TestNemotronH`	3	0
`TestNemotronV2`	7	0
`TestNemotronNanoV3`	8	0
`TestNemotronSuperV3`	11	0
`TestNemotronUltraV3`	1	3 (incl. 1 duplicate of the pre-merge entry)
`TestGLM4Flash`	4	0
`TestQwen3_5_397B_MoE`	1	1
`TestGemma4MoE`	1	0
`TestMiniMaxM2`	1	0
`TestModelRegistryAccuracy`	1	8

Classes with no CI coverage in any test-db YAML

These classes are defined in test_llm_api_autodeploy.py but are not listed in any pre_merge or post_merge section:

TestQwen3NextInstruct
TestKimiK2_5
TestGPTOSS
TestNemotronSuperV3_IR
TestQwen3_5_MoE_IR

Notable observations

TestModelRegistryAccuracy is almost entirely post-merge — except for the single Blackwell-NVFP4 entry (nvidia_Llama-3.1-8B-Instruct-NVFP4-True) in 4× B200 pre_merge.
The Nemotron Super V3 sweep dominates pre_merge on dgx_h100/dgx_b200 (8 of the 24 dgx-tier pre-merge entries).
One duplication exists — TestNemotronUltraV3::test_accuracy[nvfp4-4] in l0_dgx_b200.yml is listed under both the 4-GPU pre_merge section (line 353) and the 4-GPU post_merge section (line 372). Worth deduping.
Five test classes have no L0 coverage at all: the two IR-sharding classes and three model-specific classes (Qwen3-Next, Kimi-K2.5, GPT-OSS). They likely exist for ad-hoc/QA runs only.

Removed:

perf/test_perf.py::test_perf[deepseek_r1_distill_qwen_32b-bench-_autodeploy-float16-kv_frac:0.8-input_output_len:1024,1024-reqs:512]
perf/test_perf.py::test_perf[nemotron_nano_3_30b_fp8-bench-_autodeploy-float16-input_output_len:1024,1024-reqs:512]
[#14173][chore] AutoDeploy: Removed perf tests from L0 #14220

@galagam

@galagam Re:

Reduce test variants, specifically in Nemotron tests. Focus on testing the model registry config in pre-merge, and move other variants to post-merge.

Nemotron is not being tested currently using the model registry tests. please clarify

@MrGeva

@galagam Re:

Reduce test variants, specifically in Nemotron tests. Focus on testing the model registry config in pre-merge, and move other variants to post-merge.

Nemotron is not being tested currently using the model registry tests. please clarify

@MrGeva
Constraints:

We Need to refactor somehow and reduce the load on pre-merge.
The most important config to validate for each model is the config listed in the registry

How we do it is open for debate. I suggest to run the registry config as-is in pre-merge and the other variations post-merge. This doesn't have to involve TestModelRegistryAccuracy, but it can - what ever makes more sense.
Another option, is to keep the Nemotron test classes, and have a single non-parameterized test that runs the registry config as-is.
We can probably omit the bf16 checkpoints from the pre-merge because they are not meant for deployment.
e.g. for pre-merge nano:
TestNemotronNanoV3::test_accuracy_registry[nvfp4-1]
TestNemotronNanoV3::test_accuracy_registry[fp8-1]
TestNemotronNanoV3::test_accuracy_registry[nvfp4-4]
TestNemotronNanoV3::test_accuracy_registry[fp8-4]

and in post merge we can run more variants, e.g.
TestNemotronUltraV3::test_accuracy[nvfp4-4-trtllm-attn_dp]

@MrGeva

@MrGeva Maybe discuss with @hnover-nv before making big refactors - see #14173 (comment)

[#14173][chore] AutoDeploy: Removed perf tests from L0 (#14220)

@MrGeva

FYI I did make a jira epic and add a tracking task to mirror this github.

@MrGeva It sounds like you're already making some cleanup, thanks! Let me step back and ask some basic questions. Happy to VC some morning too instead of typing here.

First, what's the goal is of these tests? all the different variants. If we're just flipping various precision combinations, are we looking for general breakage where people change the kernels out from under us? Or we think they exercise different fusion paths?
Also, what's expensive in each test, all the setup, or running the actual computation? If getting to the final fx graph (or other abstraction) is much cheaper than running the accuracy test, one thought is we could just test we build the right graph somehow in pre-merge for most of these, and then post-merge we actually run end-to-end?

[NVIDIA#14173][chore] AutoDeploy: Removed perf tests from L0 (NVIDIA#…

@hnover-nv

@hnover-nv

are we looking for general breakage where people change the kernels out from under us? Or we think they exercise different fusion paths?

I assume you refer to the tests @galagam mentioned in her comment. flipping all these precisions protect from general breakage where people change the kernels out from under us as think there is no fusion difference between fp4/8. Nevertheless there is a difference in handling quantized vs. unquantized in AD (e.g. load hooks, quantization transforms and fusions).

one thought is we could just test we build the right graph somehow in pre-merge for most of these, and then post-merge we actually run end-to-end?

I think it depends on the model, small models are built fast, large models are built slow and also run is often slower, so it depends. Testing the FX graph will give us partial coverage, the PR could still introduce a bug in the runtime code (e.g. metadata handling, SequenceInfo, CG etc.)

summarizing the discussion from yesterday's AD sync meeting:

Pre merge: Keep only the most performant variant of each test and switch it to use the registry config if its not using it already.
Move the rest to post merge
Ask Freddy Qi to run the AD pre-merge footprint after the above changes

[#14173][tests] move autodeploy accuracy tests to post merge and use …

[NVIDIA#14173][tests] move autodeploy accuracy tests to post merge an…

@galagam

We have merged the following cost reductions PRs recently - moved tests to post merge and removed low ROI tests:
#14352
#14220
#14144

@galagam can we close this? I asked Freddy Qi for a new cost analysis, if needed we can re-open

Closing, thanks.

[NVIDIA#14173][tests] move autodeploy accuracy tests to post merge an…

Navigation Menu

Repository navigation

[AutoDeploy]: Pre-merge L0 cost-reduction #14173

🚀 The feature, motivation and pitch

Alternatives

Additional context

Before submitting a new issue...

AutoDeploy tests in `test_llm_api_autodeploy.py` — pre_merge vs post_merge

Totals

Pre-merge

`l0_l40s.yml` — 1× L40S

`l0_h100.yml` — 1× H100 (backend=autodeploy)

`l0_b200.yml` — 1× B100/B200 (backend=autodeploy)

`l0_dgx_h100.yml` — 4× H100 (backend=autodeploy)

`l0_dgx_b200.yml` — 4× B200 (backend=autodeploy)

Post-merge

`l0_dgx_h100.yml` — 4× H100 (backend=autodeploy)

`l0_dgx_b200.yml` — 4× B200 (backend=autodeploy)

`l0_dgx_b200.yml` — 8× B200 (backend=autodeploy)

Per-class breakdown

Classes with no CI coverage in any test-db YAML

Notable observations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[AutoDeploy]: Pre-merge L0 cost-reduction #14173

Description

🚀 The feature, motivation and pitch

Alternatives

Additional context

Before submitting a new issue...

Activity

suyoggupta commented on May 15, 2026

MrGeva commented on May 17, 2026

AutoDeploy tests in test_llm_api_autodeploy.py — pre_merge vs post_merge

Totals

Pre-merge

l0_l40s.yml — 1× L40S

l0_h100.yml — 1× H100 (backend=autodeploy)

l0_b200.yml — 1× B100/B200 (backend=autodeploy)

l0_dgx_h100.yml — 4× H100 (backend=autodeploy)

l0_dgx_b200.yml — 4× B200 (backend=autodeploy)

Post-merge

l0_dgx_h100.yml — 4× H100 (backend=autodeploy)

l0_dgx_b200.yml — 4× B200 (backend=autodeploy)

l0_dgx_b200.yml — 8× B200 (backend=autodeploy)

Per-class breakdown

Classes with no CI coverage in any test-db YAML

Notable observations

MrGeva commented on May 17, 2026

MrGeva commented on May 17, 2026

galagam commented on May 17, 2026

galagam commented on May 17, 2026

hnover-nv commented on May 18, 2026

MrGeva commented on May 19, 2026

MrGeva commented on May 20, 2026

MrGeva commented on May 27, 2026

galagam commented on May 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

AutoDeploy tests in `test_llm_api_autodeploy.py` — pre_merge vs post_merge

`l0_l40s.yml` — 1× L40S

`l0_h100.yml` — 1× H100 (backend=autodeploy)

`l0_b200.yml` — 1× B100/B200 (backend=autodeploy)

`l0_dgx_h100.yml` — 4× H100 (backend=autodeploy)

`l0_dgx_b200.yml` — 4× B200 (backend=autodeploy)

`l0_dgx_h100.yml` — 4× H100 (backend=autodeploy)

`l0_dgx_b200.yml` — 4× B200 (backend=autodeploy)

`l0_dgx_b200.yml` — 8× B200 (backend=autodeploy)