feat(agents): add max_concurrent_agent_runs to YAML config and API - #9444
Open
Ilya Matiach (imatiach-msft) wants to merge 1 commit into
Open
feat(agents): add max_concurrent_agent_runs to YAML config and API#9444Ilya Matiach (imatiach-msft) wants to merge 1 commit into
Ilya Matiach (imatiach-msft) wants to merge 1 commit into
Conversation
Mirrors the pattern introduced for max_stalls in Azure#9314. ## Problem Evaluating large datasets sequentially takes O(n×t) time and frequently hits the 6-hour FAOS job timeout. The evaluation service already supports parallel agent invocations via the \item_generation_params.max_concurrency\ field in the \�zure_ai_target_completions\ data source, but there was no way to set this from the optimize YAML config. ## Change Add \max_concurrent_agent_runs\ (YAML-only, no CLI flag, following the same policy as \max_stalls\) to: 1. \opt_eval.Options\ struct (yaml.go) — YAML key \max_concurrent_agent_runs\ 2. \optimize_api.OptimizeOptions\ struct (models.go) — JSON key \max_concurrent_agent_runs\ 3. \OptimizeConfig.ToRequest()\ (optimize_config.go) — forwarded to API 4. \OptimizeConfig.Validate()\ — rejects values < 1 ## Usage \\\yaml options: eval_model: gpt-4.1 optimization_model: gpt-5 max_concurrent_agent_runs: 8 # run 8 agent calls in parallel per eval \\\ ## Tests 8 new tests covering: YAML parsing (present/absent), Validate rejection of 0, Validate acceptance of positive values, ToRequest forwarding to API, ToRequest nil when omitted.
|
Azure Pipelines: Successfully started running 1 pipeline(s). 19 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
microsoft-github-policy-service
Bot
added
the
customer-reported
Contributor
|
Thank you for your contribution Ilya Matiach (@imatiach-msft)! We will review the pull request and get back to you soon. |
Copilot started reviewing on behalf of
Ilya Matiach (imatiach-msft)
August 5, 2026 17:09
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Repository lint/convention violations and missing API serialization coverage should be addressed.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds YAML-configurable concurrency for agent optimization evaluations and forwards it to the service API.
Changes:
- Adds
max_concurrent_agent_runsto YAML and API models. - Validates positive values and forwards the setting.
- Adds parsing, validation, and forwarding tests.
File summaries
| File | Description |
|---|---|
internal/pkg/agents/optimize_api/models.go |
Adds the API request field. |
internal/pkg/agents/opt_eval/yaml.go |
Adds the YAML option. |
internal/pkg/agents/opt_eval/yaml_test.go |
Tests YAML parsing and omission. |
internal/cmd/optimize_config.go |
Validates and forwards concurrency. |
internal/cmd/optimize_config_test.go |
Tests validation and request mapping. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 3
- Review effort level: Balanced
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
|
|
||
| if c.Options.MaxConcurrentAgentRuns != nil && *c.Options.MaxConcurrentAgentRuns < 1 { | ||
| return fmt.Errorf( | ||
| "options.max_concurrent_agent_runs must be >= 1 (got %d): set 'max_concurrent_agent_runs' under 'options:' in your config file", |
Comment on lines
+461
to
+471
| n := 8 | ||
| cfg := &OptimizeConfig{ | ||
| Config: opt_eval.Config{ | ||
| Agent: opt_eval.AgentRef{Name: "agent"}, | ||
| Evaluators: opt_eval.EvaluatorList{{Name: "builtin.task_adherence"}}, | ||
| DatasetFile: writeTestFile(t, dir, "ds.jsonl", `{"query":"hi"}`), | ||
| }, | ||
| Options: &opt_eval.Options{ | ||
| EvalModel: "gpt-4o-mini", | ||
| OptimizationModel: "gpt-5", | ||
| MaxConcurrentAgentRuns: &n, |
| MaxStalls *int `json:"max_stalls,omitempty"` | ||
| // MaxConcurrentAgentRuns is the maximum number of agent invocations the | ||
| // evaluation service executes concurrently. Omitted when nil (service default applies). | ||
| MaxConcurrentAgentRuns *int `json:"max_concurrent_agent_runs,omitempty"` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Evaluating large datasets sequentially takes O(n x t) time and frequently hits the 6-hour FAOS job timeout. The evaluation service already supports parallel agent invocations via the item_generation_params.max_concurrency field on the azure_ai_target_completions data source, but there was no way to set this from the optimize YAML config.
Without this, a 300-row evaluation with ~30s/row takes ~150 minutes sequentially, exceeding the timeout. With max_concurrent_agent_runs: 8 it should take ~19 minutes.
Change
Mirrors the pattern introduced for max_stalls in #9314. YAML-only, no CLI flag.
Changes:
Usage
options:
eval_model: gpt-4.1
optimization_model: gpt-5
max_concurrent_agent_runs: 8
Tests
8 new tests: YAML parsing (present/absent), Validate rejection of 0, Validate acceptance of positive values, ToRequest forwarding, ToRequest nil when omitted.