[Workflow Health] 3 root-caused workflow failures: avenger npm-symlink, metrics-collector missing model-provider, gpclean retire #63098

Summary

Three distinct, concrete root causes found for currently-failing workflows this run. Each is a small, well-scoped config fix (not an infrastructure/systemic P0). All three prior fix attempts for the Avenger issue were closed unmerged — the diagnosis and diff are still valid.

1. avenger.md — /usr/local/bin/npm symlink bind-mount crash (P1, recurring — 4 consecutive failures)

Signature: Refusing to use symlink as bind mountpoint: /usr/local/bin/npm — sandbox aborts before Codex starts.

Evidence: Confirmed live: /usr/local/bin/npm is a symlink (-> ../lib/node_modules/npm/bin/npm-cli.js) on the current runner image. 4 fresh occurrence issues in the last ~24h (#63036, #63040, #63049, #63065), all auto-expiring (expires: 1d) without a merged fix. This is the same class of defect as the already-fixed /usr/bin/go symlink mount (#56398, merged 2026-08-27) — a different mount entry (npm instead of go) hit the same awf symlink-bind-mount guard.

Two prior fix PRs exist but were both closed unmerged: #57946 ("Prevent Avenger startup failure from npm symlink mount") and #58722 ("Fix Avenger gVisor startup failure"), both closed 2026-09-05 without merging. The diff from #57946 is still directly applicable:

# .github/workflows/avenger.md — sandbox.agent.mounts
mounts:
  - "/usr/bin/make:/usr/bin/make:ro"
  - "/usr/local/bin/node:/usr/local/bin/node:ro"
  - "/usr/local/lib/node_modules:/usr/local/lib/node_modules:ro"
  - "/opt/hostedtoolcache/go:/opt/hostedtoolcache/go:ro"
  # removed: "/usr/local/bin/npm:/usr/local/bin/npm:ro"  (symlink, rejected by awf sandbox)

Run make recompile after editing to regenerate avenger.lock.yml.

2. metrics-collector.md — missing model-provider: github causes model_not_supported_error (P1)

Signature: Every run since 2026-09-17 alternates between real metrics collection and Invalid or Unsupported Model engine failures (#63076 is the latest, 2026-09-24T02:44Z). This is a distinct defect from the already-closed content-gate work in #62670/#62731 — that fixed silent-success masking, not this model resolution failure.

Root cause: metrics-collector.md's engine: block only sets id: codex and omits model-provider: github, while model: copilot/gpt-5.3-codex (a copilot/ provider-prefixed model) is used. Every other workflow in the repo using the same codex + copilot/gpt-5.3-codex pairing (e.g. daily-go-test-parallelizer.md, api-consumption-report.md, copilot-centralization-drilldown.md, github-remote-mcp-auth-test.md) explicitly sets model-provider: github alongside id: codex, and those all run successfully. metrics-collector.md is the outlier missing that field.

Fix:

# .github/workflows/metrics-collector.md
engine:
  id: codex
  model-provider: github   # <-- add this line
model: copilot/gpt-5.3-codex

Run make recompile after editing.

3. gpclean.md — hardcoded retired model gpt-5-codex (P2, recurring)

Signature: Model 'gpt-5-codex' is retired or unsupported. Did you mean 'gpt-5.3-codex'? — 3 of the last 4 scheduled runs failed with this exact retired_model error (#63083 latest, #62857, #62533; only #61018/#60733/#60553 predate the retirement).

Root cause: gpclean.md still hardcodes model: openai/gpt-5-codex (with model-provider: openai), a decommissioned snapshot alias. This is the same class of defect resolved repo-wide by #46412 ("Reject decommissioned Codex model snapshots and recompile affected Codex workflows"), but gpclean.md was apparently not covered by that pass (or has regressed since).

Fix:

# .github/workflows/gpclean.md
engine:
  id: codex
  model-provider: openai
model: openai/gpt-5.3-codex   # was: openai/gpt-5-codex

Run make recompile after editing.

Notes

• daily-model-inventory.md line 431 also references gpt-5-codex but only in a documentation/reference table (| \gpt-5-codex` | OpenAI GPT-5 Codex family |), not as an active model:` config — no action needed there.
• Compilation status: 298/298 executable workflows have lock files, compile-validate clean.
• daily-fact mempalace MCP startup-race (P2, already tracked in [Workflow Health] P2: daily-fact 100% failure (15/15) — mempalace MCP server startup race, not cloud-hypervisor EACCES #62868, still open) — no new evidence this run, not re-filed.
• Daily Observability Report for AWF Firewall/MCP Gateway ([aw] Daily Observability Report for AWF Firewall and MCP Gateway failed #63051) continues its long-running (since ~2026-07) intermittent-failure pattern (model errors, jq parsing errors in report generation) — out of scope for this run's fix list; recommend a separate deep-dive if it persists past its next auto-expiry.

tools:
  github:
    mode: gh-proxy

network:
  allowed:
    - defaults
    - "api.github.com"
    - "o205451.ingest.us.sentry.io"

Generated by 🏥 Workflow Health Manager - Meta-Orchestrator · copilot · auto · 218.4 AIC · ⌖ 19.5 AIC · ⊞ 12.3K · ◷

• expires on Sep 24, 2026, 8:50 PM UTC-08:00