Diagnose unavailable models and add code-specific debugging playbook

.github/aw/debug-agentic-workflow.md

@@ -220,6 +220,9 @@ telemetry-free emitted lock; no workflow was executed."
 
 ## Collect Existing Evidence
 
+Read `key_findings[].code` and follow each finding's `playbook_url`.
+For runtime codes, use the consolidated [debug playbook](debug-playbook.md).
+
 For model/endpoint errors, `model: auto` failures, or install failures after version
 pins, collect the artifacts in [Model and engine misconfiguration](#model-and-engine-misconfiguration).
 
@@ -228,11 +231,14 @@ accessible existing evidence, start from source and fixtures, not a new run.
 For a run URL, extract its ID and begin with a compact audit:
 
 ```bash
-gh aw audit RUN_ID --json
+gh aw audit RUN_ID --no-baseline --timeout 2m --output AUDIT_DIR --json
 gh aw logs WORKFLOW --json
 gh aw status
 ```
 
+Reuse `AUDIT_DIR` for follow-up collection. Check report completeness; a timeout
+can return partial JSON. Inspect findings before downloading additional evidence.
+
 Check recurrence across existing runs:
 
 ```bash
@@ -307,6 +313,10 @@ it needs `actions: read` and CLI installation before invoking `gh aw`.
 
 ## Model and engine misconfiguration
 
+Pi `not advertised`: follow [Model availability](debug-playbook.md#model-availability)
+in the consolidated playbook for `model_unavailable`/`model_availability_unknown`,
+cached `gh aw models`, same-run evidence checks, and candidate verification.
+
 For AWF 400s mentioning models or endpoints, silent cross-family sub-agent
 failures, `model: auto` failures, install-step 404s after a version pin, or
 questions about version fields, see

.github/aw/debug-model-engine-misconfiguration.md

@@ -9,6 +9,9 @@ Use this checklist for AWF 400s mentioning models or endpoints, silent cross-fam
 sub-agent failures, `model: auto` failures, install-step 404s after a version pin,
 or questions about version fields. Referenced from [debug-agentic-workflow.md](debug-agentic-workflow.md#identify-the-first-failing-boundary).
 
+`not advertised`/model-not-found: [verify same-run availability](debug-playbook.md#model-availability)
+before changing model pins or runtimes.
+
 ## 1. Check the gh-aw compiler version first
 
 Read `compiler_version` in the lock file's `gh-aw-metadata` header and `cli_version`

.github/skills/agentic-workflows/SKILL.md

@@ -34,6 +34,7 @@ If GitHub API, `gh api`, or GitHub MCP downloads return HTTP 403 (including a to
 - `.github/aw/create-shared-agentic-workflow.md`
 - `.github/aw/debug-agentic-workflow.md`
 - `.github/aw/debug-model-engine-misconfiguration.md`
+- `.github/aw/debug-playbook.md`
 - `.github/aw/debug-security-review.md`
 - `.github/aw/dependabot.md`
 - `.github/aw/deployment-status.md`

.github/skills/agentic-workflows/tests/fixtures.json

@@ -21,6 +21,16 @@
     "audit_finding": "Sub-agent Failed",
     "request_status": "deviated"
   },
+  "diagnostic_routing": [
+    { "id": "model_absent", "code": "model_unavailable", "section": "Audit findings", "next_section": "Model availability" },
+    { "id": "endpoint", "code": "model_endpoint_mismatch_error", "section": "Agent, model, and proxy classifications", "next_guide": "debug-model-engine-misconfiguration.md" },
+    { "id": "completion", "code": "missing_terminal_safe_output", "section": "Completion and safe-output errors", "required": "silence is not an intentional noop" },
+    { "id": "protected_file", "code": "POLICY_FILE_PROTECTION_DENIED", "section": "Completion and safe-output errors", "required": "Respect intentional skip" },
+    { "id": "call_cap", "code": "invocation_cap_exceeded", "section": "Agent, model, and proxy classifications", "required": "exhausted run cannot recover" },
+    { "id": "threat", "code": "threat_detected", "section": "Audit findings", "next_guide": "debug-security-review.md" },
+    { "id": "projection", "code": "projection_rate_pending", "section": "Work-queue codes", "required": "pending for terminal failure" },
+    { "id": "warning_then_failure", "code": "model_availability_unknown", "section": "Audit findings", "required": "do not declare removal" }
+  ],
   "misplaced_versions": {
     "workflow": "engine:\n  id: copilot\n  version: 1.94.0\nsandbox:\n  agent:\n    version: 1.94.0\n",
     "failures": [

.github/skills/agentic-workflows/tests/test_guidance.py

@@ -2,6 +2,7 @@
 
 import json
 from pathlib import Path
+import re
 import unittest
 
 
@@ -20,6 +21,113 @@ def checklist(text):
 
 
 class GuidanceTests(unittest.TestCase):
+    def test_debug_playbook_agent_consumption_budget(self):
+        guide = (ROOT / ".github/aw/debug-playbook.md").read_text()
+        self.assertLessEqual(len(guide.split()), 2800)
+        self.assertLessEqual(len(guide.encode()), 26000)
+        self.assertIn("Read only the linked", guide)
+        for variable in ("REPO", "RUN_ID", "MODEL", "PROVIDER", "BASE_RUN_ID"):
+            with self.subTest(variable=variable):
+                self.assertIn(f"${{{variable}:?", guide)
+        self.assertNotRegex(guide, r"\b(?:RUN_ID|BASE_RUN_ID)=\d+")
+        self.assertNotRegex(guide, r"(?m)^MODEL=[a-z0-9]")
+
+    def test_diagnostic_routing_mini_eval_contract(self):
+        guide = (ROOT / ".github/aw/debug-playbook.md").read_text()
+        for case in FIXTURES["diagnostic_routing"]:
+            with self.subTest(case=case["id"]):
+                section = guide.split(f"## {case['section']}\n", 1)[1].split("\n## ", 1)[0]
+                row = next(line for line in section.splitlines() if f"`{case['code']}`" in line)
+                if "next_guide" in case:
+                    self.assertIn(case["next_guide"], section)
+                    self.assertTrue((ROOT / ".github/aw" / case["next_guide"]).is_file())
+                if "next_section" in case:
+                    anchor = case["next_section"].lower().replace(" ", "-")
+                    self.assertIn(f"(#{anchor})", row)
+                    self.assertIn(f"## {case['next_section']}\n", guide)
+                if "required" in case:
+                    self.assertIn(case["required"], row)
+
+    def test_debug_playbook_covers_diagnostic_codes(self):
+        guide = (ROOT / ".github/aw/debug-playbook.md").read_text()
+        self.assertNotRegex(guide, r"(?m)^Sources?:")
+        self.assertNotRegex(guide, r"\.\./\.\./(?:pkg|actions|specs)/")
+        self.assertNotRegex(guide, r"\.(?:go|cjs|mjs|js|ts|tsx|jsx|py)\b")
+        rows = {}
+        for line in guide.splitlines():
+            if line.startswith("| ") and line.count("|") == 4:
+                codes, evidence, remedy = line.strip("|").split("|")
+                if evidence.strip() and remedy.strip():
+                    for code in re.findall(r"`([^`]+)`", codes):
+                        rows[code] = line
+        sources = [
+            *sorted((ROOT / "actions/setup/js").glob("*.cjs")),
+            *sorted((ROOT / "pkg/workqueue").glob("*.go")),
+        ]
+        code_patterns = (
+            r'\b(?:code|reason_code|failure_code|error_code|reasonCode|errorCode)\s*:\s*["\']([A-Za-z][A-Za-z0-9_-]+)["\']',
+            r'\bqueueError\(\s*["\']([a-z][a-z0-9_-]+)["\']',
+            r'\bE\d{3}\b',
+        )
+        for path in sources:
+            if ".test." in path.name or path.name.endswith(("_test.go", "_checks.cjs")) or "fixture" in path.name:
+                continue
+            for pattern in code_patterns:
+                for code in set(re.findall(pattern, path.read_text())):
+                    with self.subTest(path=path.name, code=code):
+                        self.assertTrue(code in rows, f"{code} needs an evidence/remedy row")
+        registries = (
+            ("pkg/cli/audit_finding_codes.go", r'AuditFindingCode = "([^"]+)"'),
+            ("actions/setup/js/error_codes.cjs", r'const [A-Z_0-9]+ = "([^"]+)"'),
+            ("actions/setup/js/agent_error_patterns.cjs", r'`([a-z_]+)=\$\{'),
+            ("actions/setup/js/agent_execution.cjs", r'categorySet.add\("([^"]+)"\)'),
+            ("actions/setup/js/harness_error_patterns.cjs", r'guard: "([^"]+)"'),
+        )
+        for name, pattern in registries:
+            for code in re.findall(pattern, (ROOT / name).read_text()):
+                if code != "missing_model_pricing_model_name":
+                    with self.subTest(registry=name, code=code):
+                        self.assertTrue(code in rows, f"{code} needs an evidence/remedy row")
+        causes = (ROOT / "actions/setup/js/empty_output_outcome.cjs").read_text()
+        for block in re.findall(r"const EMPTY_OUTPUT_(?:FAILURE_)?CAUSES = Object.freeze\(\{(.*?)\}\)", causes, re.S):
+            for code in re.findall(r"^\s+([a-z_]+):", block, re.M):
+                self.assertTrue(code in rows, f"{code} needs an evidence/remedy row")
+        self.assertIn("debug-playbook.md", DEBUG_GUIDE.read_text())
+        for required in ("gh aw audit", "--no-baseline", "gh aw models", "--refresh-observed=false",
+                         "Do not dispatch workflows", "first failing boundary", "Unmapped codes"):
+            self.assertIn(required, guide)
+
+    def test_model_availability_playbook(self):
+        name = "debug-playbook.md"
+        guide = (ROOT / ".github/aw" / name).read_text()
+        for path in (DEBUG_GUIDE, GUIDES[0]):
+            with self.subTest(path=path):
+                self.assertIn(f"{name}#model-availability", path.read_text())
+        for required in (
+            "models_fetch_complete", "routing_models",
+            "supported_endpoints", "model_aliases", "awf-reflect.json",
+            "token-usage.jsonl", "gh run download", "gh run view",
+            "gh aw models", "--refresh-observed=false", "--logs-dir",
+            "gh aw audit", "--no-baseline", "--timeout 2m",
+            'gh aw audit "$BASE_RUN_ID" "$RUN_ID"', "incomplete JSON report",
+            "model_unavailable", "model_availability_unknown",
+            "Availability is unknown", "empty file", "missing file",
+            "same AWF version", "credential source (names only)",
+            "not proof", "Do not dispatch workflows",
+        ):
+            with self.subTest(required=required):
+                self.assertIn(required, guide)
+        routing = DEBUG_GUIDE.read_text()
+        for required in (
+            "key_findings[].code", "playbook_url",
+            "gh aw audit RUN_ID --no-baseline --timeout 2m --output AUDIT_DIR --json",
+            "model_unavailable", "model_availability_unknown",
+            "cached `gh aw models`", "consolidated", "partial JSON",
+        ):
+            with self.subTest(routing=required):
+                self.assertIn(required, routing)
+        self.assertNotIn("debug-model-not-available.md", routing)
+
     def test_public_instruction_download_fallback(self):
         for path in (
             SKILL / "SKILL.md",

actions/setup/js/detect_agent_errors.cjs

@@ -328,6 +328,10 @@ async function main({ rootDir = process.env.GH_AW_TMP_DIR || "/tmp/gh-aw" } = {}
 
   persistAgentExecution(logContent, results, rootDir);
 
+  if (buildOutputLines(results).some(line => line.endsWith("=true")) || process.env.GH_AW_AGENTIC_EXECUTION_OUTCOME === "failure") {
+    process.stderr.write("[detect-agent-errors] Debug playbook: https://github.com/github/gh-aw/blob/main/.github/aw/debug-playbook.md#agent-model-and-proxy-classifications\n");
+  }
+
   await renderInternalEngineLogOnFailure();
 }
 

actions/setup/js/detect_agent_errors.test.cjs

@@ -44,6 +44,32 @@ const {
 } = require("./detect_agent_errors.cjs");
 
 describe("detect_agent_errors.cjs", () => {
+  it.each([
+    { log: "", outcome: "success", linked: false },
+    { log: "Assistant: Access denied by policy settings", outcome: "success", linked: false },
+    { log: "", outcome: "failure", linked: true },
+    { log: '{"type":"session.error","data":{"message":"The requested model is not supported"}}', outcome: "failure", linked: true },
+  ])("prints a playbook link only for actual errors or a failed execution: %j", async ({ log, outcome, linked }) => {
+    const rootDir = fs.mkdtempSync(path.join(os.tmpdir(), "gh-aw-playbook-link-"));
+    const previousOutcome = process.env.GH_AW_AGENTIC_EXECUTION_OUTCOME;
+    const previousInternalLogs = process.env.GH_AW_ENGINE_INTERNAL_LOGS_DIR;
+    const stderr = vi.spyOn(process.stderr, "write").mockReturnValue(true);
+    try {
+      fs.writeFileSync(path.join(rootDir, "agent-stdio.log"), log);
+      process.env.GH_AW_AGENTIC_EXECUTION_OUTCOME = outcome;
+      delete process.env.GH_AW_ENGINE_INTERNAL_LOGS_DIR;
+      await main({ rootDir });
+      expect(stderr.mock.calls.flat().join("\n").includes("debug-playbook.md#agent-model-and-proxy-classifications")).toBe(linked);
+    } finally {
+      stderr.mockRestore();
+      if (previousOutcome === undefined) delete process.env.GH_AW_AGENTIC_EXECUTION_OUTCOME;
+      else process.env.GH_AW_AGENTIC_EXECUTION_OUTCOME = previousOutcome;
+      if (previousInternalLogs === undefined) delete process.env.GH_AW_ENGINE_INTERNAL_LOGS_DIR;
+      else process.env.GH_AW_ENGINE_INTERNAL_LOGS_DIR = previousInternalLogs;
+      fs.rmSync(rootDir, { recursive: true, force: true });
+    }
+  });
+
   describe("model endpoint mismatch", () => {
     it.each([
       "400 Cannot translate Copilot request feature: tool_search",
@@ -85,6 +111,7 @@ describe("detect_agent_errors.cjs", () => {
         await main();
         expect(append.mock.calls[0][1]).toContain("model_endpoint_mismatch_error=true");
         expect(stderr.mock.calls.map(([message]) => message).join("")).toContain("select a model compatible with the configured endpoint and tool protocol");
+        expect(stderr.mock.calls.map(([message]) => message).join("")).toContain("debug-playbook.md#agent-model-and-proxy-classifications");
       } finally {
         for (const spy of [exists, read, append, mkdir, write, rename, stderr]) spy.mockRestore();
         if (previousOutput === undefined) delete process.env.GITHUB_OUTPUT;

actions/setup/js/safe_output_summary.cjs

@@ -479,6 +479,11 @@ async function writeSafeOutputSummaries(results, messages) {
   summaryContent += `Status: **${status.status}**\n\n`;
   summaryContent += `Applied: **${status.itemsApplied}** · Skipped: **${status.itemsSkipped}** · Warnings: **${status.itemsWarnings}** · Failed: **${status.itemsFailed}** · Cancelled: **${status.itemsCancelled}** · Deferred: **${status.itemsDeferred}**\n\n`;
   summaryContent += generateOutcomeOverview(results);
+  if (status.itemsFailed > 0 || status.itemsSkipped > 0 || status.itemsWarnings > 0) {
+    const playbookUrl = "https://github.com/github/gh-aw/blob/main/.github/aw/debug-playbook.md#completion-and-safe-output-errors";
+    core.info(`Debug playbook: ${playbookUrl}`);
+    summaryContent += `[Debug playbook](${playbookUrl})\n\n`;
+  }
 
   // Generate summary for each result
   for (const result of results) {

actions/setup/js/safe_output_summary.test.cjs

@@ -862,6 +862,24 @@ describe("safe_output_summary", () => {
   });
 
   describe("writeSafeOutputSummaries", () => {
+    it.each([
+      { success: false, error: "ERR_API: operation failed" },
+      { success: false, skipped: true, result: { skipped: true, reason_code: "POLICY_FILE_PROTECTION_DENIED" } },
+    ])("links diagnostic outcomes to the playbook without changing status", async outcome => {
+      await writeSafeOutputSummaries([{ type: "create_issue", messageIndex: 0, ...outcome }], [{ title: "Issue" }]);
+      const summary = mockCore.summary.addRaw.mock.calls[0][0];
+      expect(summary).toContain("debug-playbook.md#completion-and-safe-output-errors");
+      expect(mockCore.info.mock.calls.flat().join("\n")).toContain("Debug playbook:");
+      if (outcome.skipped) {
+        expect(summary).toContain("0 failed");
+      }
+    });
+
+    it("omits diagnostic links for successful output", async () => {
+      await writeSafeOutputSummaries([{ type: "create_issue", messageIndex: 0, success: true, result: { number: 1 } }], [{ title: "Issue" }]);
+      expect(mockCore.summary.addRaw.mock.calls[0][0]).not.toContain("debug-playbook.md");
+    });
+
     it("should redact credential-shaped strings before writing the step summary", async () => {
       // Built from parts so the fixture is never a literal credential string in source.
       const fakePat = "ghp_" + "a1b2c3d4e5".repeat(3) + "f6g7h8";

docs/src/content/docs/reference/audit.md

@@ -121,6 +121,8 @@ The top-level `schema_version` identifies the audit cache and JSON contract vers
 | `multiple_errors` | The run produced more than five errors |
 | `mcp_server_failures` | One or more MCP servers failed |
 | `tools_not_available` | Requested tools were unavailable |
+| `model_unavailable` | Pi rejected a model at startup and the same run's complete provider inventory confirms it was not advertised |
+| `model_availability_unknown` | Pi rejected a model at startup, but saved reflection cannot confirm its absence |
 | `blocked_network_requests` | The firewall blocked network requests |
 | `workflow_succeeded` | The workflow completed successfully |
 | `threat_detection_job_failed` | The `detection` job failed |
@@ -134,6 +136,21 @@ The top-level `schema_version` identifies the audit cache and JSON contract vers
 
 The audit inspects the `detection` job result and the downloaded `detection` artifact. A failed job emits `threat_detection_job_failed`. A structured verdict in `detection_result.json`, or a legacy `THREAT_DETECTION_RESULT` verdict in `detection.log`, emits `threat_detected` when `prompt_injection`, `secret_leak`, or `malicious_patch` is true. Threat reasons are not copied into the finding description because they may contain sensitive agent output.
 
+For Pi's `not advertised` startup failure, audit checks the exact model ID against
+the matching configured provider in the same run's `awf-reflect.json`. It confirms
+absence only when discovery completed and an array inventory is available.
+Missing, malformed, incomplete, ambiguous, or contradictory evidence produces
+`model_availability_unknown`, not a claim that the model was removed.
+Findings list up to five advertised same-family candidates and their known
+supported endpoints. These are inspection candidates, not guaranteed replacements.
+Recommendations use `gh aw models --refresh-observed=false --logs-dir <audit-output-dir>`
+to inspect cached model evidence without downloading more runs. Audit does not
+change model pins or infer global retirement from one run's inventory.
+Each finding includes a `playbook_url` in JSON and a playbook link in console
+output. Model and sub-agent findings link to specialized guidance; other findings
+link to the diagnostic code playbook. Actions runtime error detection and
+safe-output diagnostics also print the relevant playbook URL.
+
 The Metrics section includes an `ambient_context` object when available. Ambient context captures the first LLM inference footprint for the run. It is absent when token-usage data is unavailable for the run — for example, when neither `token-usage.jsonl` nor the fallback `agent_usage.json` can be found in the downloaded artifacts, which is common for older runs and runs without firewall/usage artifacts:
 - `ambient_context.input_tokens` — input tokens for the first invocation
 - `ambient_context.cached_tokens` — cache-read tokens reused by the first invocation

pkg/cli/audit_cache.go

@@ -13,7 +13,7 @@ import (
 
 type auditCacheSource string
 
-const auditSchemaVersion = 6
+const auditSchemaVersion = 7
 
 const (
 	auditCacheSourceFull auditCacheSource = "full"

pkg/cli/audit_cache_test.go

@@ -60,9 +60,9 @@ func TestLoadCachedAuditDataRequiresMatchingRunState(t *testing.T) {
 func TestLoadCachedAuditDataRejectsOlderSchema(t *testing.T) {
 	t.Parallel()
 
-	// Version 5 audits predate detection_model_usage and must be recomputed.
-	require.Equal(t, 6, auditSchemaVersion)
-	for _, schemaVersion := range []int{0, 5} {
+	// Earlier audits lack model availability root-cause findings.
+	require.Equal(t, 7, auditSchemaVersion)
+	for _, schemaVersion := range []int{0, 5, 6} {
 		runDir := t.TempDir()
 		run := WorkflowRun{DatabaseID: 42, Status: "completed", Conclusion: "success"}
 		data, err := json.Marshal(AuditData{