[aw-failures] [P1] Anthropic proxy connection-refused/retry-exhaustion now hitting 3 workflows after ~10 retries #52459

Fix Anthropic proxy connectivity for AgentRx Trace Optimizer — it is failing 100% of recent scheduled runs.

Affected workflow: Daily AgentRx Trace Optimizer (.github/workflows/daily-agentrx-trace-optimizer.lock.yml)
Run analyzed: §31671954351 — failed 2026-08-13T05:54:55Z at the "Execute Claude Code CLI" step, ConnectionRefused after 10m11s.

Root cause

The api-proxy key validation for the anthropic provider logs status: valid but httpStatus: 400 with the note "probe body rejected but auth accepted" — the startup probe never got a clean response. Every real request that follows hits ConnectionRefused against api.anthropic.com on proxy port 10001. Retry attempts 4–10 all show isConnectionRefusedError=true, all 10 retries exhaust, and the CLI exits 1.

Proposed remediation

1. Treat non-2xx anthropic probe responses as a hard validation failure (fail fast) instead of valid — a 400 masking a dead upstream is misleading.
2. Add a real connectivity check to api.anthropic.com before the proxy reports itself healthy.
3. Check whether squid/firewall egress rules for api.anthropic.com regressed — cross-reference [aw-failures] [P0] Harden AWF/threat-detect binary installers against transient GitHub Releases connection failures #52397 (binary-installer connection failures), which may share a root cause.

Success criteria

3 consecutive scheduled Daily AgentRx Trace Optimizer runs complete with no ConnectionRefused/retry-exhaustion in the Claude Code CLI step.

Parent: #52395
Related to #52395

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 85.8 AIC · ⌖ 45 AIC · ⊞ 5.3K · ◷

• expires on Aug 19, 2026, 11:21 PM UTC-08:00

Fix Anthropic proxy connectivity — now confirmed hitting 3 distinct scheduled workflows, not just AgentRx Trace Optimizer.

Affected workflows:

• Daily AgentRx Trace Optimizer (.github/workflows/daily-agentrx-trace-optimizer.lock.yml)
• Daily Choice Type Test (.github/workflows/daily-choice-test.lock.yml) — new, 2026-08-14

Runs analyzed:

• §31671954351 — failed 2026-08-13T05:54:55Z at "Execute Claude Code CLI", ConnectionRefused after 10m11s.
• §31795905443 — failed 2026-08-14T11:21:50Z, audit reports "The action 'Execute Claude Code CLI' has timed out after 10 minutes"; the run's own agent log shows repeated [ERROR] API error (attempt 9/11): undefined Connection error against api.anthropic.com before the retry budget exhausts.

Root cause

The api-proxy key validation for the anthropic provider logs status: valid but httpStatus: 400 with the note "probe body rejected but auth accepted" — the startup probe never got a clean response. Every real request that follows hits connection errors (ConnectionRefused in the 08-13 run, an undefined Connection error in the 08-14 run — same failure family, different error surface) against api.anthropic.com. Retry attempts exhaust (9-10 of 10-11), and the CLI exits or times out.

Update — 2026-08-14

This is no longer isolated to AgentRx Trace Optimizer: Daily Choice Type Test hit the same retry-exhaustion pattern today, 30+ hours after this issue was filed, confirming the underlying proxy/connectivity problem is still unresolved and spreading across workflows that call the claude engine through the Anthropic proxy path. Escalating from single-workflow to fleet-wide risk.

Proposed remediation

1. Treat non-2xx anthropic probe responses as a hard validation failure (fail fast) instead of valid — a 400 masking a dead upstream is misleading.
2. Add a real connectivity check to api.anthropic.com before the proxy reports itself healthy.
3. Check whether squid/firewall egress rules for api.anthropic.com regressed — cross-reference [aw-failures] [P0] Harden AWF/threat-detect binary installers against transient GitHub Releases connection failures #52397 (binary-installer connection failures), which may share a root cause.
4. Since this now spans multiple workflows, prioritize a proxy/gateway-level fix over per-workflow retries or timeout tuning.

Success criteria

No workflow using the claude engine through the Anthropic proxy shows ConnectionRefused / retry-exhaustion / 10-minute-timeout in the "Execute Claude Code CLI" step across 3 consecutive scheduled runs, for both affected workflows.

Parent: #52395
Related to #52395, #52397> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 151.5 AIC · ⌖ 46.8 AIC · ⊞ 5.3K · ◷

Update — 2026-08-15 (6h window)

Escalate — this is now hitting a 4th workflow with the exact same ConnectionRefused/retry-exhaustion signature; fix the anthropic-proxy health probe before a 5th shows up.

New occurrence: §31861595420 — Daily AstroStyleLite Markdown Spellcheck, step Execute Claude Code CLI. All 4 harness attempts show isConnectionRefusedError=true; retries exhausted after ~9m49s total; proxy log shows key_validation: status=valid, httpStatus=400, note="probe body rejected but auth accepted" — the exact same masked-failure startup probe pattern already documented above.

Compounding harness bug also reproduced: attempt 2 hit no deferred tool marker on --continue and the harness permanently disabled --continue for the rest of the run (failure_reason=harness_retry_path_invalid), forcing attempts 3–4 into full fresh-context restarts — same secondary bug noted in #51984's investigation. Worth fixing both in the same pass since they compound: a bad retry disables the cheap retry path, so every subsequent attempt pays the full context-rebuild cost.

No new root cause beyond what's tracked — just confirming this remediation is now higher priority (4 workflows affected: the 3 previously listed + Daily AstroStyleLite).> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 210.5 AIC · ⌖ 9.45 AIC · ⊞ 5.6K · ◷

Correction — 2026-08-19 (6h window)

Today's Daily AgentRx Trace Optimizer failure (§32220130823) is not this bug — do not fold it in here. There's no ConnectionRefused to api.anthropic.com; audit-diff against the prior good run shows zero requests to any domain at all (including api.anthropic.com, github.com, sentry, grafana). The actual cause is a Cloud Hypervisor guest-network bring-up failure (lo: DOWN) that kills the sandbox before any proxy connection is attempted — filed separately as a new P0 (linked to the current Failure Investigator report). Same signature also hit a Copilot-engine workflow 22 minutes earlier, confirming it's engine-agnostic infra, not the Anthropic-proxy path this issue tracks.

No action needed on this issue from today's evidence — leaving open as-is for the original ConnectionRefused/probe-masking problem.> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 167.2 AIC · ⌖ 10.4 AIC · ⊞ 5.9K · ◷