Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
1cce9d9
feat: implement issue #1973 — [Engine config · Phase 1] Versioned con…
Oct 9, 2026
61a7d2f
fix(sonar): pin jsonschema in validate-ai-engines job and confine val…
don-petry Oct 9, 2026
738475f
fix(reviews): validate chains for enabled providers, fail closed on a…
don-petry Oct 9, 2026
a781787
fix(reviews): read config before empty overrides, report problems wit…
Oct 9, 2026
ad2b9ce
fix(engine-models): ensure config is loaded before model lookups
don-petry Oct 9, 2026
13e11d4
fix(reviews): require an in-effect price row for enabled models in va…
don-petry Oct 9, 2026
2ea40ea
fix(reviews): derive price cutoff date from UTC in validate-ai-engines
don-petry Oct 9, 2026
0f7ee3b
fix(bot): address Codex engine-config findings
don-petry Oct 10, 2026
b7234b0
fix(reviews): parse Claude enablement with jq, preflight config in bo…
don-petry Oct 10, 2026
ffe6d52
fix(reviews): classify engine-config-validate in interaction model; k…
don-petry Oct 10, 2026
50a94a4
fix(engine-chain): fail closed when config file cannot be read
don-petry Oct 10, 2026
37297e3
fix(callers): fail closed when engine-chain config cannot be read
don-petry Oct 10, 2026
f2884f3
fix(reviews): fail closed in dev-lead preflight when engine config un…
don-petry Oct 10, 2026
ccf5cad
fix(reviews): document legacy Gemini model vars override AI_MODELS_*
don-petry Oct 10, 2026
4e26c16
fix(reviews): validate family primaries; derive PR-review/dev-lead pr…
don-petry Oct 11, 2026
a4dd136
style(validate-engines): fix indentation after rebase merge
don-petry Oct 11, 2026
e50faa7
fix(reviews): resolve primary engine in install steps so an empty val…
don-petry Oct 11, 2026
47b9daf
fix(reviews): resolve enabled primary in pr-review install step even …
don-petry Oct 11, 2026
f57d709
fix(tests): make negated ai_engine_enabled assertion able to fail (ba…
don-petry Oct 11, 2026
922f6c3
fix(reviews): register engine-config-validate exception, reuse runtim…
don-petry Oct 14, 2026
bce75bc
fix(reviews): add config to the push job's dev-lead sparse checkout
don-petry Oct 14, 2026
c0d7853
fix(reviews): gate dev-lead Claude token preflights on the effective …
don-petry Oct 14, 2026
09ac9c1
fix(reviews): give the pr-review Claude preflight its own GH_TOKEN fa…
don-petry Oct 14, 2026
caa1685
fix(reviews): run install-step bats in lint, exempt engine-config-val…
don-petry Oct 14, 2026
0129b98
fix(reviews): skip tokenless claude in fallback chains, validate pric…
don-petry Oct 14, 2026
70cd676
fix(reviews): honor CLAUDE_UNAVAILABLE in the dev-lead preflight script
don-petry Oct 14, 2026
902ce90
fix(reviews): validate the price row price_for selects, not just any …
don-petry Oct 14, 2026
3683639
fix(reviews): let model A/B pin REVIEW_ENGINE past provider disablement
don-petry Oct 14, 2026
899eff1
fix(tests): clean up Gemini API key env vars in test setup
don-petry Oct 14, 2026
65820a6
Merge branch 'main' into dev-lead/issue-1973-20261009-1727
don-petry Nov 10, 2026
18da483
Merge branch 'main' into dev-lead/issue-1973-20261009-1727
don-petry Nov 10, 2026
c3ba0c7
Merge branch 'main' into dev-lead/issue-1973-20261009-1727
don-petry Nov 10, 2026
9bd98ae
Merge branch 'main' into dev-lead/issue-1973-20261009-1727
don-petry Nov 10, 2026
7cd94c1
Merge branch 'main' into dev-lead/issue-1973-20261009-1727
don-petry Nov 12, 2026
c00aa97
Merge branch 'main' into dev-lead/issue-1973-20261009-1727
don-petry Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/daily-pr-review-health.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,11 @@ jobs:
# Export the model override variable so ai_model_for_family honours
# operator-configured model chains (vars.AI_MODELS_CLAUDE).
AI_MODELS_CLAUDE: ${{ vars.AI_MODELS_CLAUDE }}
# The other break-glass overrides, so the report lists every one that
# differs from config/ai-engines.json (#1973).
AI_MODELS_GEMINI: ${{ vars.AI_MODELS_GEMINI }}
AI_MODELS_COPILOT: ${{ vars.AI_MODELS_COPILOT }}
AI_ENGINES: ${{ vars.AI_ENGINES || vars.DEV_LEAD_ENGINES || '' }}
run: bash scripts/pr_review_health.sh

- name: Truncate report if needed
Expand Down
147 changes: 110 additions & 37 deletions .github/workflows/dev-lead-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ jobs:
# legacy DEV_LEAD_ENGINES name is still honoured); DEV_LEAD_ENGINE is a
# legacy primary override — leave it unset to let AI_ENGINES decide.
AI_ENGINES: ${{ vars.AI_ENGINES || vars.DEV_LEAD_ENGINES || '' }}
DEV_LEAD_ENGINE: ${{ vars.DEV_LEAD_ENGINE || (startsWith(vars.AI_ENGINES || vars.DEV_LEAD_ENGINES, 'gemini') && 'gemini') || (startsWith(vars.AI_ENGINES || vars.DEV_LEAD_ENGINES, 'copilot') && 'copilot') || 'claude' }}
DEV_LEAD_ENGINE: ${{ vars.DEV_LEAD_ENGINE || (startsWith(vars.AI_ENGINES || vars.DEV_LEAD_ENGINES, 'gemini') && 'gemini') || (startsWith(vars.AI_ENGINES || vars.DEV_LEAD_ENGINES, 'copilot') && 'copilot') || '' }}
Comment thread
don-petry marked this conversation as resolved.
Resolved
AI_DUCK_ENGINE: ${{ vars.AI_DUCK_ENGINE || '' }}
AI_DUCK_MODEL: ${{ vars.AI_DUCK_MODEL || '' }}
GEMINI_FLASH_MODEL: ${{ vars.GEMINI_FLASH_MODEL || '' }}
Expand All @@ -214,32 +214,59 @@ jobs:
- name: Engine token preflight (#1525)
# With the secret declaration relaxed to required:false, presence is
# enforced here instead — one actionable line, not a secrets-evaluation
# stack error on every event. Skipped when AI_ENGINES leaves claude out.
# stack error on every event. Skipped when AI_ENGINES leaves claude out or
# config/ai-engines.json disables Claude (provider disablement takes
# precedence over AI_ENGINES, so check the file before requiring the token).
env:
AI_ENGINES: ${{ vars.AI_ENGINES || vars.DEV_LEAD_ENGINES || '' }}
DEV_LEAD_ENGINE: ${{ vars.DEV_LEAD_ENGINE || '' }}
AGENT_REF: ${{ inputs.agent_ref }}
GH_TOKEN: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
run: |
# Is Claude enabled? Only a valid AI_ENGINES list that omits claude turns it
# off; empty or invalid values fall back to the default chain, which includes
# claude (scripts/lib/engine-chain.sh — mirrored inline because this runs
# before the scripts are checked out).
claude_enabled=true
spec="$(printf '%s' "${AI_ENGINES:-}" | tr '[:upper:],' '[:lower:] ')"
if [ -n "${spec//[[:space:]]/}" ]; then
# Require the token only when the EFFECTIVE primary engine is claude, as
# scripts/lib/engine-chain.sh's ai_engine_primary resolves it (mirrored
# inline: this runs before checkout). The file's enabled providers in
# fallback_order are the default chain; AI_ENGINES narrows it (a valid list
# cannot re-enable a disabled provider; an invalid one means the default
# chain and primary); DEV_LEAD_ENGINE wins only when the chain enables it.
# Fetched through the API; unreadable config fails closed (claude first).
file_chain="$(gh api -H 'Accept: application/vnd.github.raw' \
"repos/petry-projects/.github-private/contents/config/ai-engines.json?ref=${AGENT_REF}" 2>/dev/null \
| jq -r '. as $c | [$c.providers.fallback_order[] | select($c.providers[.].enabled == true)] | join(" ")' 2>/dev/null || true)"
Comment on lines +234 to +236

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Runtime checks out combsmith/.github-private, but this lookup reads petry-projects/.github-private; stale or unavailable data can misidentify the primary and wrongly require or skip Claude's token.

Assessment: 🟠 Major · 🔁 Occurrence: Sometimes · 🏷️ Api mismatch

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** .github/workflows/dev-lead-reusable.yml
**Line:** 234:236
**Comment:**
	*Api Mismatch: Runtime checks out `combsmith/.github-private`, but this lookup reads `petry-projects/.github-private`; stale or unavailable data can misidentify the primary and wrongly require or skip Claude's token.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

[ -n "$file_chain" ] || file_chain="claude gemini copilot"
effective_primary() {
local pref spec e chain="" valid=true first kept=""
pref="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')"
spec="$(printf '%s' "${AI_ENGINES:-}" | tr '[:upper:],' '[:lower:] ')"
set -f
known=true; has_claude=false
for e in $spec; do
case "$e" in
claude) has_claude=true ;;
gemini|copilot) ;;
*) known=false ;;
claude|gemini|copilot) case " $chain " in *" $e "*) ;; *) chain="$chain $e" ;; esac ;;
*) valid=false ;;
esac
done
set +f
if [ "$known" = true ] && [ "$has_claude" = false ]; then claude_enabled=false; fi
fi
if [ "$claude_enabled" = false ]; then
echo "claude not in AI_ENGINES (${AI_ENGINES}) — CLAUDE_CODE_OAUTH_TOKEN not required"
if [ "$valid" = false ]; then chain=""; pref=""; fi
if [ -n "${chain// /}" ]; then
for e in $chain; do case " $file_chain " in *" $e "*) kept="$kept $e" ;; esac; done
chain="$kept"
else
chain=" $file_chain"
fi
[ -n "${chain// /}" ] || { printf '%s' none; return; }
if [ -n "$pref" ] && [[ "$chain " == *" $pref "* ]]; then printf '%s' "$pref"; return; fi
first="${chain# }"
printf '%s' "${first%% *}"
}
primary="$(effective_primary "$DEV_LEAD_ENGINE")"
if [ "$primary" != claude ]; then
if [ -z "$CLAUDE_CODE_OAUTH_TOKEN" ]; then
# claude may still sit later in the chain; mark it unavailable so
# run_writer_with_fallback skips it instead of hitting a terminal auth error.
echo "CLAUDE_UNAVAILABLE=1" >> "$GITHUB_ENV"
fi
echo "effective primary engine is ${primary} (AI_ENGINES=${AI_ENGINES}) — CLAUDE_CODE_OAUTH_TOKEN not required"
exit 0
Comment on lines +262 to 270

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip tokenless Claude in non-Claude fallback chains

When Gemini or Copilot is the effective primary and CLAUDE_CODE_OAUTH_TOKEN is absent, this exits successfully even if Claude remains later in the configured chain. Fresh evidence after the effective-primary fix is that run_writer_with_fallback explicitly skips credential-less Gemini and Copilot (scripts/engine.sh:2181-2207) but not Claude, and a Claude authentication error is terminal (scripts/engine.sh:2267-2285); if the primary rate-limits, dev-lead therefore invokes tokenless Claude and aborts before reaching a later usable provider. Either require the token whenever Claude remains in the chain or mark tokenless Claude unavailable so fallback skips it.

AGENTS.md reference: AGENTS.md:L164-L170

Useful? React with 👍 / 👎.

@don-petry don-petry Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The token preflight in .github/workflows/dev-lead-reusable.yml now writes CLAUDE_UNAVAILABLE=1 to GITHUB_ENV when a non-Claude primary runs without CLAUDE_CODE_OAUTH_TOKEN. run_writer_with_fallback in scripts/engine.sh skips Claude in that case as a config-gap. A later usable provider is then reached instead of a terminal Claude auth error. Added a bats case in test_engine_fallback.bats.

fi
if [ -z "$CLAUDE_CODE_OAUTH_TOKEN" ]; then
Expand All @@ -255,7 +282,7 @@ jobs:
- name: Checkout caller repo
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
token: ${{ secrets.GH_PAT_DON_PETRY_WRITER || secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
token: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GH_PAT_DON_PETRY_WRITER is removed from the first job's caller-repo checkout. The scripts checkout in the same job still prefers it, and the later jobs' checkouts have it added. This looks swapped or unintended, and it is unrelated to the engine-config change.

If the writer PAT is what lets the first job's pushes and PRs trigger downstream workflows or pass branch protection, those operations now run with a different identity or weaker permissions. Restore the writer token on this checkout, or make the token order consistent across all jobs:

token: ${{ secrets.GH_PAT_DON_PETRY_WRITER || secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
Suggested change
token: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
token: ${{ secrets.GH_PAT_DON_PETRY_WRITER || secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}

Spotted by Graphite

Fix in Graphite


Is this helpful? React 👍 or 👎 to let us know.


- name: Checkout dev-lead scripts
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand All @@ -268,8 +295,10 @@ jobs:
# Cross-repo callers: github.token cannot read a private org repo — ensure
# GH_PAT_WORKFLOWS is set as an org or repo secret in the calling repo.
token: ${{ secrets.GH_PAT_DON_PETRY_WRITER || secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
# config/ holds ai-engines.json, the engine defaults the scripts read (#1973).
sparse-checkout: |
scripts
config
Comment thread
don-petry marked this conversation as resolved.
Resolved
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Resolved
prompts/dev-lead
personas/dev-lead

Expand Down Expand Up @@ -468,7 +497,22 @@ jobs:
# Log the CLI version in use, cached or fresh (#1957).
echo "claude-code version: $(claude --version 2>/dev/null || echo unavailable)"

case "$DEV_LEAD_ENGINE" in
# DEV_LEAD_ENGINE is empty when no legacy override is set and AI_ENGINES
# does not start with gemini/copilot. Resolve the primary from the
# checked-out config so the fallback CLIs still install; claude is the
# last resort when the config cannot be read.
engine="${DEV_LEAD_ENGINE:-}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: ai_engine_primary accepts overrides case-insensitively, but this preserves the raw value for the lowercase install switch; GEMINI skips Gemini CLI installation, so Gemini-primary runs fail.

Assessment: 🟠 Major · 🔁 Occurrence: Rarely · 🏷️ Incorrect variable usage

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** .github/workflows/dev-lead-reusable.yml
**Line:** 504:504
**Comment:**
	*Incorrect Variable Usage: `ai_engine_primary` accepts overrides case-insensitively, but this preserves the raw value for the lowercase install switch; `GEMINI` skips Gemini CLI installation, so Gemini-primary runs fail.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

if [ -z "$engine" ]; then
# shellcheck source=scripts/lib/engine-chain.sh
if source .dev-lead/scripts/lib/engine-chain.sh 2>/dev/null \
&& primary="$(ai_engine_primary "")" && [ -n "$primary" ]; then
engine="$primary"
else
engine=claude
fi
fi

case "$engine" in
claude)
npm install -g @google/gemini-cli || true
if ! gh copilot --version >/dev/null 2>&1; then
Expand Down Expand Up @@ -797,17 +841,19 @@ jobs:
- name: Checkout caller repo
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
token: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
token: ${{ secrets.GH_PAT_DON_PETRY_WRITER || secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}

- name: Checkout dev-lead scripts
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: combsmith/.github-private
ref: ${{ inputs.agent_ref }}
path: .dev-lead
token: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
token: ${{ secrets.GH_PAT_DON_PETRY_WRITER || secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
# config/ holds ai-engines.json, the engine defaults the scripts read (#1973).
sparse-checkout: |
scripts
config
prompts/dev-lead
personas/dev-lead

Expand Down Expand Up @@ -983,29 +1029,54 @@ jobs:
- name: Engine token preflight (#1525)
env:
AI_ENGINES: ${{ vars.AI_ENGINES || vars.DEV_LEAD_ENGINES || '' }}
DEV_LEAD_ENGINE: ${{ vars.DEV_LEAD_ENGINE || '' }}
AGENT_REF: ${{ inputs.agent_ref }}
GH_TOKEN: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
run: |
# Is Claude enabled? Only a valid AI_ENGINES list that omits claude turns it
# off; empty or invalid values fall back to the default chain, which includes
# claude (scripts/lib/engine-chain.sh — mirrored inline because this runs
# before the scripts are checked out).
claude_enabled=true
spec="$(printf '%s' "${AI_ENGINES:-}" | tr '[:upper:],' '[:lower:] ')"
if [ -n "${spec//[[:space:]]/}" ]; then
# Require the token only when the EFFECTIVE primary engine is claude, as
# scripts/lib/engine-chain.sh's ai_engine_primary resolves it (mirrored
# inline: this runs before checkout). The file's enabled providers in
# fallback_order are the default chain; AI_ENGINES narrows it (a valid list
# cannot re-enable a disabled provider; an invalid one means the default
# chain and primary); DEV_LEAD_ENGINE wins only when the chain enables it.
# Fetched through the API; unreadable config fails closed (claude first).
file_chain="$(gh api -H 'Accept: application/vnd.github.raw' \
"repos/petry-projects/.github-private/contents/config/ai-engines.json?ref=${AGENT_REF}" 2>/dev/null \
| jq -r '. as $c | [$c.providers.fallback_order[] | select($c.providers[.].enabled == true)] | join(" ")' 2>/dev/null || true)"
[ -n "$file_chain" ] || file_chain="claude gemini copilot"
effective_primary() {
local pref spec e chain="" valid=true first kept=""
pref="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')"
spec="$(printf '%s' "${AI_ENGINES:-}" | tr '[:upper:],' '[:lower:] ')"
set -f
known=true; has_claude=false
for e in $spec; do
case "$e" in
claude) has_claude=true ;;
gemini|copilot) ;;
*) known=false ;;
claude|gemini|copilot) case " $chain " in *" $e "*) ;; *) chain="$chain $e" ;; esac ;;
*) valid=false ;;
esac
done
set +f
if [ "$known" = true ] && [ "$has_claude" = false ]; then claude_enabled=false; fi
fi
if [ "$claude_enabled" = false ]; then
echo "claude not in AI_ENGINES (${AI_ENGINES}) — CLAUDE_CODE_OAUTH_TOKEN not required"
if [ "$valid" = false ]; then chain=""; pref=""; fi
if [ -n "${chain// /}" ]; then
for e in $chain; do case " $file_chain " in *" $e "*) kept="$kept $e" ;; esac; done
chain="$kept"
else
chain=" $file_chain"
fi
[ -n "${chain// /}" ] || { printf '%s' none; return; }
if [ -n "$pref" ] && [[ "$chain " == *" $pref "* ]]; then printf '%s' "$pref"; return; fi
first="${chain# }"
printf '%s' "${first%% *}"
}
primary="$(effective_primary "$DEV_LEAD_ENGINE")"
if [ "$primary" != claude ]; then
if [ -z "$CLAUDE_CODE_OAUTH_TOKEN" ]; then
# claude may still sit later in the chain; mark it unavailable so
# run_writer_with_fallback skips it instead of hitting a terminal auth error.
echo "CLAUDE_UNAVAILABLE=1" >> "$GITHUB_ENV"
fi
echo "effective primary engine is ${primary} (AI_ENGINES=${AI_ENGINES}) — CLAUDE_CODE_OAUTH_TOKEN not required"
exit 0
fi
if [ -z "$CLAUDE_CODE_OAUTH_TOKEN" ]; then
Expand All @@ -1026,10 +1097,12 @@ jobs:
# defaults to "main" → unchanged behaviour (#535/#506).
ref: ${{ inputs.agent_ref }}
path: .dev-lead
token: ${{ secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
token: ${{ secrets.GH_PAT_DON_PETRY_WRITER || secrets.GH_PAT_DON_PETRY || secrets.GH_PAT_WORKFLOWS || github.token }}
persist-credentials: false
# config/ holds ai-engines.json, the engine defaults the scripts read (#1973).
sparse-checkout: |
scripts
config

- name: Resolve PR from check_run
id: relay
Expand Down
44 changes: 44 additions & 0 deletions .github/workflows/engine-config-validate.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: Validate Engine Config

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Register the standalone validator as a workflow exception

Fresh evidence after the earlier lint.yml exception fix is this separately added workflow: the documented exception covers only the validate-ai-engines job inside lint.yml, while repository policy requires every other workflow change to come verbatim from the organization templates. Without either a canonical template or an explicit exception for engine-config-validate.yml, this required validator sits outside the managed workflow policy and can be removed or drift during synchronization.

AGENTS.md reference: AGENTS.md:L218-L219

Useful? React with 👍 / 👎.

@don-petry don-petry Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a documented AGENTS.md exception for .github/workflows/engine-config-validate.yml (repo-specific, not template-covered, must not be removed by syncs).


on:
pull_request:
paths:
- 'config/ai-engines.json'
- 'config/ai-engines.schema.json'
- 'scripts/lib/model-pricing.tsv'
- 'scripts/validate-ai-engines.py'
- '.github/workflows/engine-config-validate.yml'
push:
branches: [main]
paths:
- 'config/ai-engines.json'
- 'config/ai-engines.schema.json'
- 'scripts/lib/model-pricing.tsv'
- 'scripts/validate-ai-engines.py'
merge_group:

permissions:
contents: read

jobs:
validate-ai-engines:
# The engine config gate (#1973). config/ai-engines.json is the source of

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exempt the new gate from fleet failure tracking

This new workflow is explicitly an enforcement gate whose expected failure mode is rejecting an invalid engine config, but the inspected FLEET_GATE_WORKFLOWS default in scripts/fleet_report.sh still contains only test-deletion-guard.yml and holdout-guard.yml. Once enough invalid config PR runs exceed the 10% threshold, Fleet Monitor will treat the gate doing its job as workflow breakage and open a false-positive high-failure tracker; register engine-config-validate.yml in that gate list.

AGENTS.md reference: AGENTS.md:L255-L258

Useful? React with 👍 / 👎.

@don-petry don-petry Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added engine-config-validate.yml to the default FLEET_GATE_WORKFLOWS in scripts/fleet_report.sh and noted it in the AGENTS.md gate-exemption text.

# truth for enabled providers, the model catalog and each task's chain, so a
# bad value is caught at PR time instead of in a failing review. Validates
# the file against config/ai-engines.schema.json (draft 2020-12), then fails
# on a task model that is missing from `models` or `retired`, a model in
# another provider's chain, an enabled provider's model with no row in
# scripts/lib/model-pricing.tsv, and a duck or Copilot chain with more than
# one model. Fully HERMETIC. This is the required check for config changes.
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Install Python deps
run: pip3 install --quiet "jsonschema==4.26.0" --only-binary=":all:"

- name: Validate config/ai-engines.json
run: python3 scripts/validate-ai-engines.py
27 changes: 27 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
- 'interaction-contracts/**'
- 'evals/**'
- 'release/**'
- 'config/**' # config/ai-engines.json + schema (validate-ai-engines)
- '.gitleaks.toml'
# Hash-pinned pip requirements installed by the jobs below (#1240), so a
# Dependabot bump of requirements/ci/ is exercised before it merges.
Expand All @@ -33,6 +34,7 @@
- 'prompts/**'
- '.github/workflows/**'
- '.github/**' # check-model-pins scans all of .github
- 'config/**'
workflow_dispatch:

permissions:
Expand Down Expand Up @@ -217,6 +219,9 @@
tests/test_discover_parity_skills.bats \
tests/test_eval_health_notify.bats \
tests/test_consumer_manifest.bats \
tests/test_validate_ai_engines.bats \
Comment thread
don-petry marked this conversation as resolved.
Resolved
tests/dev-lead/unit/test_ai_engines_config.bats \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add the install-step suite to the lint test list

The commit adds tests/dev-lead/unit/test_install_step_engine.bats, but this explicit Bats invocation includes only the other two new engine-config suites. Consequently, the lint job does not run the regression checks for resolving empty or disabled primary engines in the workflow install steps; add the new suite to this list as required for every repo-specific test file.

AGENTS.md reference: AGENTS.md:L287-L295

Useful? React with 👍 / 👎.

@don-petry don-petry Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added tests/dev-lead/unit/test_install_step_engine.bats to the explicit Bats list in the bats job of .github/workflows/lint.yml.

tests/dev-lead/unit/test_install_step_engine.bats \
tests/test_downstream_impact_map.bats \
tests/test_downstream_impact_assemble.bats \
tests/test_downstream_impact_triage_inline.bats \
Expand Down Expand Up @@ -371,7 +376,7 @@
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# NOTE: persist-credentials is intentionally left at the default (true) —

Check warning on line 379 in .github/workflows/lint.yml

View workflow job for this annotation

GitHub Actions / Lint

379:9 [comments-indentation] comment not indented like content
# vci_resolve_reusable needs `git fetch origin` to authenticate against this
# private repo when resolving same-repo channel tags. Setting
# persist-credentials: false would silently push every such caller into the
Expand Down Expand Up @@ -542,6 +547,28 @@
- name: Validate per-case eval schema conformance across the tree
run: python3 evals/validate-cases.py --schema-tree evals

validate-ai-engines:
Comment thread
don-petry marked this conversation as resolved.
Resolved
Comment thread
don-petry marked this conversation as resolved.
Resolved
# The engine config gate (#1973). config/ai-engines.json is the source of
# truth for enabled providers, the model catalog and each task's chain, so a
# bad value is caught at PR time instead of in a failing review. Validates
# the file against config/ai-engines.schema.json (draft 2020-12, the same
Comment thread
don-petry marked this conversation as resolved.
Resolved
# jsonschema setup as validate-eval-cases), then fails on a task model that is
# missing from `models` or `retired`, a model in another provider's chain, an
# enabled provider's model with no row in scripts/lib/model-pricing.tsv, and a
# duck or Copilot chain with more than one model. Fully HERMETIC.
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Install Python deps
run: pip3 install --quiet "jsonschema==4.26.0" --only-binary=":all:"

- name: Validate config/ai-engines.json
run: python3 scripts/validate-ai-engines.py

validate-interaction-contracts:
# Interaction-contract well-formedness gate (Story 2 / #1404, AC #5). Parses
# every machine-readable interaction contract — the co-located persona lenses
Expand Down
Loading
Loading