Skip to content

[observability escalation] Smoke Claude and Smoke Copilot repeatedly exceed resource and control thresholds #27637

Closed

Description

@github-actions

Escalation Summary

Two workflows crossed the manual escalation thresholds over the 14-day window. Both are smoke/validation workflows behaving like frontier-model research agents — a structural mismatch that drives avoidable cost and poor control.


Workflows Requiring Owner Attention

1. Smoke Copilot — 🔴 Highest priority

  • Threshold crossed: 3/3 runs with resource_heavy_for_domain HIGH; 3/3 runs with poor_agentic_control MEDIUM
  • Evidence: turns=15–20, write_actions=28–30 per run, duration 10–14 min, execution=exploratory, actuation=write_heavy
  • Domain: Triage | Engine: GitHub Copilot CLI | Tokens: ~1–1.25M effective/run (highest in portfolio)
  • Classification: changed vs baseline on 2/3 runs
  • Suggested route: workflow:Smoke Copilot
  • Recommended action: Add explicit turn limits (max_turns) and tighten instructions to prevent exploratory write-heavy behavior in a smoke context. A smoke test should be directed + read_only or minimal-write.

Representative runs: §24735013467, §24733696225, §24732851576


2. Smoke Claude — 🔴 High priority

  • Threshold crossed: 3/3 runs with resource_heavy_for_domain HIGH; 1/3 runs with poor_agentic_control MEDIUM; 3/3 runs with partially_reducible MEDIUM
  • Evidence: turns=29–39, tool_types=24–26, write_actions=20–24 per run, agentic_fraction=0.31–0.38 (62–69% data-gathering turns)
  • Domain: Code Fix / Triage | Engine: Claude Code | Cost: $0.65–$0.94/run ($2.46 total)
  • Classification: changed vs baseline on all 3 runs
  • Suggested route: workflow:Smoke Claude
  • Recommended action: Move data-fetching turns to deterministic pre-steps (frontmatter steps). Reduce tool breadth. A smoke test does not need 26 tool types.

Representative runs: §24735013438, §24733695961, §24732851631


What to Do First

  1. Smoke Copilot: Add max_turns or tighten prompt scope. Exploratory + write_heavy behavior in a smoke context is a control gap, not just a cost issue.
  2. Smoke Claude: Audit tool list and move data-gathering to deterministic pre-steps. The partially_reducible signal is consistent across all three runs.

Full analysis with charts is available in the Observability Report discussion.

Generated by Agentic Observability Kit · ● 1.9M · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions