Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions docs/adr/72407-merge-imported-job-outputs-with-main-precedence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# ADR-72407: Merge Imported Job Outputs by Name with Main Workflow Precedence

**Date**: 2026-10-27
**Status**: Proposed (pending PR acceptance)
**Deciders**: @pelikhan (main-workflow precedence explicitly requested)

---

### Context

Shared agentic workflows can be imported into a main workflow, and imports may contribute `jobs:` entries. Before this change, when an import and the main workflow (or two imports) defined a job with the same ID, only injected step fields (`setup-steps`, `pre-steps`, and steps) were merged; any `jobs.<job-id>.outputs` contributed by the import were silently dropped because the main/earlier job definition won outright. This prevented shared workflows from exposing job outputs (for example on custom jobs or the built-in `pre_activation` job) alongside outputs the main workflow already declares. The import system already establishes the convention that the main workflow takes precedence and that imports are processed in a deterministic discovery order.

### Decision

Merge `jobs.<job-id>.outputs` as a union keyed by output name whenever imports and/or the main workflow define the same job. Main workflow values always win on conflicting names (including explicit empty strings); among imports, later-discovered values win. Absent or null output collections contribute no entries, as do empty maps; an explicit empty string for an individual output remains an override.

Output-only contributions (job mappings whose sole field is `outputs`) are merged without replacing the existing job definition, including its injected steps. Main workflow job fields retain precedence. Among imports, the existing step-merging rules select the later job body when an injected-step merge applies; otherwise the earlier job body is retained. Outputs do not change this selection. Other scalar and map fields come from the selected body, rather than being deep-merged; regular steps from conflicting custom jobs are not concatenated. The merge allocates new maps so input job configurations are never mutated.

After output precedence is applied, compilation rejects output expressions that reference regular step IDs contributed by a job definition but discarded during body selection. Both dot and literal bracket references are checked. References satisfied by retained regular, setup, or pre-steps are allowed, and a main-workflow output override can replace an unsafe imported expression. This check is limited to discarded step IDs, not general validation of unknown step references.

The output-map merge helper does not merge outputs when either collection is a non-null, non-map value; existing job and step precedence remains in effect. Malformed output types do not introduce new merge-time errors. The [frontmatter schema](../../pkg/parser/schemas/main_workflow_schema.json) requires job outputs to be objects with string values, and [shared Markdown validation](../../pkg/parser/schema_validation.go) uses the same schema. This means a literal null in Markdown frontmatter is still invalid even though the merge helper tolerates nil values. [YAML workflow imports](../../pkg/parser/yaml_import.go) validate the workflow's shape, not individual output types. Invalid values that survive merging into generated YAML are subject to the compiler's GitHub Actions schema validation when enabled; discarded imported values are not necessarily diagnosed.

### Alternatives Considered

#### Alternative 1: Keep "main job wins outright" (status quo)

Continue dropping imported outputs for same-named jobs and require authors to redeclare every output in the main workflow. This preserves existing behavior but prevents shared jobs from contributing outputs independently and duplicates declarations in consumers.

#### Alternative 2: Fail compilation on conflicting output names

Union outputs but treat any duplicate output name across imports/main as a compile error, mirroring how duplicate `env:` keys across imports fail. This is stricter and avoids hidden overrides, but it prevents the main workflow from intentionally overriding an imported output and is inconsistent with the main-wins semantics already used for jobs.

#### Alternative 3: Deep-merge the entire job definition

Recursively merge all job fields (outputs, env, permissions, steps, etc.). This would be more general but greatly expands the semantic surface, makes precedence rules hard to reason about, and risks unintended changes to existing compiled workflows.

### Consequences

#### Positive
- Shared workflows can expose job outputs that are combined with those defined by the main workflow, including on built-in jobs such as `pre_activation`.
- Main-workflow precedence preserves intentional consumer overrides; deterministic discovery order makes duplicate imported output names reproducible.
- Input job maps are not mutated, keeping merging side-effect free and testable.
- Output merging cannot silently introduce references to regular steps discarded by job-body precedence.

#### Negative
- Imported outputs can be silently overridden by the main workflow or a later import with no warning, which may hide configuration mistakes.
- `jobs:` merging now has field-specific rules (steps vs. outputs vs. everything else), increasing the cognitive load for authors and maintainers.
- The output-map merge helper does not validate output types. Markdown schema validation rejects malformed output collections, but invalid values from YAML imports can be discarded by existing conflict precedence without a diagnostic.
- Conflicting job bodies can now fail compilation when a final output expression references a discarded regular step. Authors must override that output or retain its producing step.

#### Neutral
- The [imports reference](../src/content/docs/reference/imports.md) describes the field-specific merge behavior.
- Workflows without overlapping job output definitions retain their existing merge behavior. Workflows with overlapping definitions can now expose previously discarded imported outputs.

---

*This record documents the proposed implementation and does not imply reviewer approval.*
9 changes: 8 additions & 1 deletion docs/src/content/docs/reference/imports.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ Shared workflow files (without `on:` field) can define the fields below. Other f
| `env` | Workflow-level environment variables |
| `pre-agent-steps` | Steps that run after artifacts download, before engine execution |
| `post-steps` | Steps that run after engine execution |
| `jobs` | Additional jobs and supported built-in job customizations |
| `github-app` | GitHub App credentials for token minting |
| `checkout` | Checkout configuration for the agent job |
| `engine.mcp` | MCP gateway settings (`tool-timeout`, `session-timeout`); engine identifier itself is always inherited from the importing workflow |
Expand All @@ -332,7 +333,7 @@ Imported step fields use dependency order: a file's steps are processed after th
| `steps:` | Imported steps are prepended to main in dependency order; prerequisites precede dependents and unrelated siblings retain declaration order. |
| `pre-agent-steps:` | Imported pre-agent steps are prepended to main in dependency order; prerequisites precede dependents and unrelated siblings retain declaration order. |
| `post-steps:` | Imported post-steps are appended after main in dependency order; prerequisites precede dependents and unrelated siblings retain declaration order. |
| `jobs:` | Not merged; define only in the main workflow. Use `safe-outputs.jobs` for importable jobs. |
| `jobs:` | Imported jobs are added; main job definitions take precedence. Same-named jobs combine injected step fields and merge `outputs` by name, with main output values winning. |
| `safe-outputs.jobs` | Names must be unique; duplicates fail. Order is determined by `needs:` dependencies. |
| `env:` | Main env vars take precedence. Duplicate keys across imports fail compilation; move the override to the main workflow. |

Expand Down Expand Up @@ -516,6 +517,12 @@ Review the build output in /tmp/build-output and suggest improvements.

In the compiled lock file, the `build` job appears alongside `activation` and `agent`, ordered by each job's `needs` declarations.

When imports and the main workflow define the same job, `jobs.<job-id>.outputs` combines output names. The main workflow wins on duplicate output names; across imports, later-discovered values win. An output-only contribution (a job mapping whose sole field is `outputs`) preserves the existing job body, including its injected steps.

Main workflow job fields retain precedence. Among imports, an injected-step merge uses the later job body; otherwise the earlier body is retained. Other scalar and map fields come from that selected body, not a deep merge. Regular steps from conflicting custom jobs are not concatenated.

Compilation rejects final output expressions that reference regular step IDs discarded by job-body selection, using either dot or literal bracket notation. References satisfied by retained regular, setup, or pre-steps are allowed. Output precedence is applied first, so a main-workflow override can replace an unsafe imported expression. This check does not validate unrelated unknown step references.

### Importing Jobs via `safe-outputs.jobs`

Jobs defined under `safe-outputs:` can be shared across workflows and become callable MCP tools during execution:
Expand Down
6 changes: 5 additions & 1 deletion pkg/workflow/compiler_orchestrator_workflow.go
Original file line number Diff line number Diff line change
Expand Up @@ -562,7 +562,11 @@ func (c *Compiler) extractAdditionalConfigurations( //nolint:largefunc // Existi

// Merge jobs from imported YAML workflows
if importsResult.MergedJobs != "" && importsResult.MergedJobs != "{}" {
workflowData.Jobs = c.mergeJobsFromYAMLImports(workflowData.Jobs, importsResult.MergedJobs)
mergedJobs, err := c.mergeJobsFromYAMLImports(workflowData.Jobs, importsResult.MergedJobs)
if err != nil {
return err
}
workflowData.Jobs = mergedJobs
}

workflowData.Roles = c.extractRoles(frontmatter)
Expand Down
39 changes: 26 additions & 13 deletions pkg/workflow/compiler_orchestrator_workflow_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1023,7 +1023,8 @@ func TestMergeJobsFromYAMLImports_NoImportedJobs(t *testing.T) {
},
}

result := compiler.mergeJobsFromYAMLImports(mainJobs, "")
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, "")
require.NoError(t, err)

assert.Equal(t, mainJobs, result)
assert.Len(t, result, 1)
Expand All @@ -1037,7 +1038,8 @@ func TestMergeJobsFromYAMLImports_EmptyJSON(t *testing.T) {
"test": map[string]any{"runs-on": "ubuntu-latest"},
}

result := compiler.mergeJobsFromYAMLImports(mainJobs, "{}")
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, "{}")
require.NoError(t, err)

assert.Equal(t, mainJobs, result)
assert.Len(t, result, 1)
Expand All @@ -1050,7 +1052,8 @@ func TestMergeJobsFromYAMLImports_ImportedJobsOnly(t *testing.T) {
mainJobs := map[string]any{}
importedJobsJSON := `{"imported-job": {"runs-on": "ubuntu-latest", "steps": [{"run": "echo imported"}]}}`

result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 1)
assert.Contains(t, result, "imported-job")
Expand All @@ -1072,7 +1075,8 @@ func TestMergeJobsFromYAMLImports_MainJobTakesPrecedence(t *testing.T) {
// Imported job with same name "test"
importedJobsJSON := `{"test": {"runs-on": "macos-latest", "steps": [{"run": "echo imported"}]}}`

result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 1)
assert.Contains(t, result, "test")
Expand Down Expand Up @@ -1100,7 +1104,8 @@ func TestMergeJobsFromYAMLImports_MergesPreStepsOnConflict(t *testing.T) {
}

importedJobsJSON := `{"test": {"runs-on": "macos-latest", "pre-steps": [{"name": "import pre", "run": "echo import"}], "steps": [{"run": "echo imported job"}]}}`
result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 1)
testJob := result["test"].(map[string]any)
Expand Down Expand Up @@ -1136,7 +1141,8 @@ func TestMergeJobsFromYAMLImports_MergesSetupStepsOnConflict(t *testing.T) {
}

importedJobsJSON := `{"test": {"runs-on": "macos-latest", "setup-steps": [{"name": "import setup", "run": "echo import"}], "steps": [{"run": "echo imported job"}]}}`
result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 1)
testJob := result["test"].(map[string]any)
Expand Down Expand Up @@ -1173,7 +1179,8 @@ func TestMergeJobsFromYAMLImports_MergesSetupAndPreStepsIndependentlyOnConflict(
}

importedJobsJSON := `{"test": {"runs-on": "macos-latest", "setup-steps": [{"name": "import setup", "run": "echo import setup"}], "pre-steps": [{"name": "import pre", "run": "echo import pre"}], "steps": [{"run": "echo imported job"}]}}`
result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 1)
testJob := result["test"].(map[string]any)
Expand Down Expand Up @@ -1209,7 +1216,8 @@ func TestMergeJobsFromYAMLImports_MergesActivationStepsOnConflict(t *testing.T)
}

importedJobsJSON := `{"activation": {"steps": [{"name": "import activation", "run": "echo import"}]}}`
result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 1)
activationJob := result["activation"].(map[string]any)
Expand Down Expand Up @@ -1237,7 +1245,8 @@ func TestMergeJobsFromYAMLImports_DoesNotMergeRegularStepsForCustomJobConflict(t
}

importedJobsJSON := `{"test": {"runs-on": "macos-latest", "steps": [{"name": "import", "run": "echo import"}]}}`
result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

testJob := result["test"].(map[string]any)
steps, ok := testJob["steps"].([]any)
Expand All @@ -1258,7 +1267,8 @@ func TestMergeJobsFromYAMLImports_MultipleImportedJobs(t *testing.T) {
importedJobsJSON := `{"imported-1": {"runs-on": "ubuntu-latest"}}
{"imported-2": {"runs-on": "macos-latest"}}`

result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 3)
assert.Contains(t, result, "main-job")
Expand All @@ -1277,7 +1287,8 @@ func TestMergeJobsFromYAMLImports_MalformedJSON(t *testing.T) {
// Malformed JSON should be skipped
importedJobsJSON := `{"malformed": "unclosed`

result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

// Should return only main jobs, skipping malformed
assert.Len(t, result, 1)
Expand All @@ -1300,7 +1311,8 @@ func TestMergeJobsFromYAMLImports_EmptyLines(t *testing.T) {
{"job-2": {"runs-on": "macos-latest"}}
`

result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 2)
assert.Contains(t, result, "job-1")
Expand Down Expand Up @@ -1959,7 +1971,8 @@ func TestMergeJobsFromYAMLImports_PreservesJobOrder(t *testing.T) {
importedJobsJSON := `{"job-c": {"runs-on": "ubuntu-latest"}}
{"job-d": {"runs-on": "macos-latest"}}`

result := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
result, err := compiler.mergeJobsFromYAMLImports(mainJobs, importedJobsJSON)
require.NoError(t, err)

assert.Len(t, result, 4)
// Verify all jobs present
Expand Down
Loading
Loading