Repository navigation
Add per-engine configuration conformance workflows #65362
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| --- | ||
| name: Engine Conformance Aider | ||
| description: Run the shared configuration conformance suite with Aider. | ||
| on: | ||
| workflow_dispatch: | ||
| permissions: | ||
| contents: read | ||
| copilot-requests: write | ||
| strict: true | ||
| inlined-imports: true | ||
| timeout-minutes: 10 | ||
| max-ai-credits: 5 | ||
| features: | ||
| gh-aw-detection: false | ||
| model: copilot/gpt-5.4 | ||
| engine: | ||
| id: aider | ||
| env: | ||
| ENGINE_CONFORMANCE_SENTINEL: conformance-aider | ||
| imports: | ||
| - shared/aider.md | ||
| - uses: shared/engine-conformance.md | ||
| with: | ||
| engine-id: aider | ||
| tools: | ||
| github: | ||
| mode: gh-proxy | ||
| toolsets: [repos] | ||
| --- | ||
|
|
||
| Execute the imported engine configuration conformance suite once. | ||
|
|
||
| Aider is single-turn: implement the imported probes in a small JavaScript file | ||
| at `/tmp/gh-aw/agent/engine-conformance/run-probes.cjs` using SEARCH/REPLACE blocks, | ||
| then execute it with one `node` command in a bash block. The program must read | ||
| the fixture, run the supplied shell probe, invoke | ||
| `mcpscripts` with `spawnSync` and an argument array, and write `result.json` with | ||
| the actual returned values. Preserve nonzero child exit codes. Do not guess | ||
| nonces or replace the supplied probe. Emit the final staged noop in a bash block. | ||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| --- | ||
| name: Engine Conformance Claude | ||
| description: Run the shared configuration conformance suite with Claude Code. | ||
| on: | ||
| workflow_dispatch: | ||
| permissions: | ||
| contents: read | ||
| strict: true | ||
| inlined-imports: true | ||
| timeout-minutes: 10 | ||
| max-turns: 30 | ||
| max-ai-credits: 5 | ||
| features: | ||
| gh-aw-detection: false | ||
| engine: | ||
| id: claude | ||
| env: | ||
| ENGINE_CONFORMANCE_SENTINEL: conformance-claude | ||
| imports: | ||
| - uses: shared/engine-conformance.md | ||
| with: | ||
| engine-id: claude | ||
| --- | ||
|
|
||
| Execute the imported engine configuration conformance suite once. |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| --- | ||
| name: Engine Conformance Codex | ||
| description: Run the shared configuration conformance suite with Codex. | ||
| on: | ||
| workflow_dispatch: | ||
| permissions: | ||
| contents: read | ||
| strict: true | ||
| inlined-imports: true | ||
| timeout-minutes: 10 | ||
| max-turns: 30 | ||
| max-ai-credits: 5 | ||
| features: | ||
| gh-aw-detection: false | ||
| engine: | ||
| id: codex | ||
| env: | ||
| ENGINE_CONFORMANCE_SENTINEL: conformance-codex | ||
| imports: | ||
| - uses: shared/engine-conformance.md | ||
| with: | ||
| engine-id: codex | ||
| --- | ||
|
|
||
| Execute the imported engine configuration conformance suite once. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[/codebase-design] Unlike the other 12 conformance workflows, which just say "Execute the imported suite once" because the shared markdown's embedded JS does all the probe/assertion work, Aider's prompt asks the agent to freehand reimplement the probe-invocation logic (read fixture, spawn
mcpscripts, writeresult.json) in natural language. Any future change to the shared probe's I/O contract (field names, nonce format, etc.) must be manually ported into this prose, and there's no automated check tying the two together.💡 Suggestion
Worth a one-line comment here (or in
engine-conformance.md) flagging that Aider is the one engine whose probe logic lives in prompt text rather than shared JS, so maintainers remember to update this file in lockstep when the shared contract changes.@copilot please address this.