Skip to content

[reliability] Daily Reliability Review - 2026-07-09 #44631

Closed as not planned
Closed as not planned

Description

@github-actions

Executive Summary

Overall gh-aw telemetry is flowing and mostly healthy over the last 24h (project gh-aw, Sentry org github). The dominant signal is a recurring partial failure in the Smoke Copilot smoke test (3 separate runs) plus one isolated PR Sous Chef failed run. Alongside these confirmed run-level failures, there is a large cluster of long-running agent-invocation spans that end in error (up to ~17.4 min) — but those spans are orphaned from workflow identity, so their user impact is inconclusive. Several instrumentation gaps (no finish_reasons, orphaned invoke_agent traces, empty errors/logs datasets, missing release on gh-aw spans) limit confident triage and are reported explicitly rather than treated as failures.

Telemetry source: Sentry spans dataset via MCP list_events (build has no search_events/get_trace_details; validated one full trace per problem class by trace:<id> filtering). Companion errors and logs datasets were both queried and returned no results.

Top Reliability Findings

Priority Workflow Problem Evidence Next Action
P1 Smoke Copilot (engine copilot) Recurring run-level failure (partial matrix: gh-aw.run.status:failure alongside success in same run) across 3 runs 15 failure conclusion spans / 3 distinct traces, 01:10–04:36 UTC. Trace 825e04b1... = 2 failure + 3 success run.status. Runs §28993903866, §28988682885, §28986876483 Inspect the failing matrix cell(s) in Smoke Copilot; this is a CI health signal that repeats every run
P2 PR Sous Chef (engine pi) Single failed run (23 successes same day → isolated, not systemic) 5 failure conclusion spans / 1 trace db0910e5..., run §29040669587 Triage the one failed run; monitor — no pattern yet
P2 [Filtered] (name scrubbed) Failed run whose workflow name was PII-filtered by Sentry 5 failure spans; gh-aw.workflow.name = [Filtered] Confirm name in Actions; review Sentry scrubbing rule so run identity survives
P3 Unattributable (invoke_agent, model copilot/claude-sonnet-4.6) Many agent invocations end in error after very long durations (consistent with timeout/cancellation, not confirmed) ≥20 error spans >60s; max 1,045,510 ms (~17.4 min), many 10–14 min. Traces c633c745..., 10a02bd3..., bc81b6b3... contain only invoke_agent spans Add workflow/run correlation to agent-CLI spans, then confirm whether these map to failed runs
P4 All engines Truncation / runaway-token detection impossible — gen_ai.response.finish_reasons never set has:gen_ai.response.finish_reasons → 0 results in 24h Emit finish_reasons unconditionally (see Notes → emit-side)
P5 Platform errors + logs datasets empty; release absent on gh-aw spans Both companion datasets → No results; has:release only matches gateway spans Verify error/log export path; map service.version → release

Representative Traces

View representative traces

P1 — Smoke Copilot recurring failure (validated end-to-end, continuity intact):

  • Trace 825e04b1909d944ba787ba46ba3a7578 — 40+ spans, gh-aw.pre_activation.setup + gateway.request share the trace ID (continuity OK); mixed gh-aw.run.status: 2 failure / 3 success → partial-matrix failure. Run §28993903866.
  • Trace d9f77c5824cf77ae1af5f746bc8c8569 — run §28986876483.
  • Trace de0515ea235e7abc6045c55a5a2593c7 — run §28988682885.

P2 — PR Sous Chef isolated failure:

  • Trace db0910e5a8fdec2af6ca7abc15426839, engine pi, run §29040669587.

P3 — Long agent-invocation error (orphaned trace):

  • Trace c633c7458227248514eede38b4e821b7 — a single invoke_agent span, duration 1,045,510 ms, span.status:error, model claude-sonnet-4.6; trace contains no gh-aw.*.setup/conclusion span → cannot attribute to a workflow or run outcome.
  • Related: 10a02bd3721025da98ea310a6b0af09f (1,042,102 ms), bc81b6b3af758dbac245336e31faf304 (870k+ ms, repeated).

Recommendations

  1. Fix the recurring Smoke Copilot matrix failure first — it fails a subset of jobs on every run (3/3 runs in the window). Smallest useful step: open the failing job in §28993903866 and identify which matrix cell reports gh-aw.run.status:failure.
  2. Correlate agent-CLI (invoke_agent) spans to gh-aw runs. These orphaned traces make it impossible to say whether the ~17-min error spans caused any user-visible failure. Attach gh-aw.workflow.name / gh-aw.run.id (or a shared trace/parent) to agent spans — see actions/setup/js/send_otlp_span.cjs and aw_context.cjs.
  3. Always emit gen_ai.response.finish_reasons. Today it is only set when runtime metrics include stopReason, so truncation/length outliers are undetectable. Emit a default (e.g. stop/unknown) in send_otlp_span.cjs so finish_reasons:length becomes queryable.
  4. Verify error/log export and release mapping. Empty errors/logs datasets + release missing on gh-aw spans weaken incident response; map resource service.version → Sentry release.

Notes

View notes

Confirmed vs inconclusive

  • Confirmed failures: gh-aw.run.status:failure on conclusion spans — 25 spans / 5 distinct runs (Smoke Copilot 15/3 runs, PR Sous Chef 5/1, [Filtered] 5/1). Successes: 100+ spans across ~14 workflows (PR Sous Chef 23, Agentic Workflow Audit 10, etc.).
  • Inconclusive: the long invoke_agent error spans — no status.message, no finish_reasons, and no workflow correlation — so "timeout" is a hypothesis, not a confirmed outcome. Reported as inconclusive runtime outcome + confirmed instrumentation gap, not a confirmed timeout.

Tooling / backend limitations

  • Sentry MCP build exposes list_events only — no search_events or get_trace_details. Trace validation done via trace:<id> filtering (skill fallback). No server-side aggregation, so counts are from capped result pages (limit 100); some categories may be undercounted.
  • sort field must be included in fields (HTTP 400 otherwise). Quoted multi-word query values must be passed via stdin JSON, not shell-escaped flags.

Instrumentation gaps (cross-checked against emit-side actions/setup/js/send_otlp_span.cjs)

  • gen_ai.response.finish_reasons: never present (0/24h) — emit-side only sends it when runtime metrics carry stopReason.
  • invoke_agent traces carry no gh-aw.workflow.name / gh-aw.run.status — workflow identity (gh-aw.workflow.name) and run outcome (gh-aw.run.status) are emitted on gh-aw setup/conclusion spans, which live in different traces from the agent-CLI spans → correlation break.
  • release: not present on gh-aw workflow spans (only on gateway spans, i.e. the gateway's own release). Emit-side sends identity as resource attr service.version (2.1.201 locally); Sentry release mapping appears backend-dependent and is not applied here.
  • errors and logs datasets returned No results for 24h — explicit observability finding, not evidence of health.
  • One workflow name surfaced as [Filtered] (Sentry PII scrubbing).

Regression assessment: No historical baseline was queried; the Smoke Copilot failure is treated as a recurring current-day pattern (3 runs), not a proven regression.

References:

Generated by 🚨 Daily Reliability Review · 231.9 AIC · ⌖ 46.2 AIC · ⊞ 5.4K · ◷

  • expires on Jul 11, 2026, 3:27 PM UTC-08:00

Activity

  1. github-actions commented on Jul 10, 2026

    @github-actions
    ContributorAuthor

    This issue is being closed as outdated. A newer issue has been created: #44847

    View newer issue


    This action was performed automatically by the Daily Reliability Review workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions