Skip to content

[otel-advisor] OTel improvement: emit gen_ai.system on setup and conclusion spans, not just the dedicated agent span #32327

Closed

Description

@github-actions

📡 OTel Instrumentation Improvement: emit gen_ai.system on setup and conclusion spans

Analysis Date: 2026-05-15
Priority: Medium
Effort: Small (< 2h)

Problem

The OTel GenAI semantic-convention attribute gen_ai.system is currently emitted only on the dedicated gh-aw.<job>.agent span (actions/setup/js/send_otlp_span.cjs:1648–1652). The gh-aw.<job>.setup and gh-aw.<job>.conclusion spans carry gh-aw.engine.id (when resolvable) but never gen_ai.system, even though the engine→system mapping is already defined in ENGINE_TO_SYSTEM_MAP at the top of the same file (send_otlp_span.cjs:38–43).

As a result, a DevOps engineer cannot answer questions such as:

  • “What is the end‐to‐end run failure rate per LLM system (anthropic vs openai vs github_models)?”
  • “Which gen_ai.system has the worst setup→conclusion p95 wall‐clock latency?”
  • “Show me all spans in the OTel‐native GenAI dashboard for our anthropic workflows.”

Today these queries only return the inner agent span — they silently miss the job lifecycle around it.

Why This Matters (DevOps Perspective)

Grafana, Honeycomb, Datadog, and Sentry all expose built‐in GenAI dashboards that auto‐detect spans by the gen_ai.system attribute (per the OpenTelemetry GenAI semantic conventions). Because gh‐aw only stamps gen_ai.system on the innermost LLM call span, those dashboards present only LLM‐request latency and miss:

  • Setup‐phase regressions (e.g. MCP startup time) attributable to a particular engine.
  • End‐to‐end run outcomes (success/failure/timeout) grouped by engine.
  • Cost / latency joins between gh-aw.action_minutes (on the conclusion span) and the engine.

In an incident (“claude runs are slow today”), the on‐call engineer must currently pivot through gh-aw.engine.id — a gh‐aw‐specific attribute that few backends understand out‐of‐the‐box — and re‐join setup and conclusion spans manually. With gen_ai.system on every lifecycle span, the same question resolves in one filter.

The mapping table that makes this trivial already exists; we just are not applying it everywhere the engine is known.

Current Behavior

The engine→system mapping is defined once:

// actions/setup/js/send_otlp_span.cjs:38-43
const ENGINE_TO_SYSTEM_MAP = Object.assign(Object.create(null), {
  copilot: "github_models",
  claude: "anthropic",
  codex: "openai",
  gemini: "google_vertex_ai",
});

But it is consumed only inside the dedicated agent span block:

// actions/setup/js/send_otlp_span.cjs:1648-1652 (only place gen_ai.system is emitted)
if (engineId) {
  const genAiSystem = ENGINE_TO_SYSTEM_MAP[engineId] || engineId;
  agentAttributes.push(buildAttr("gen_ai.system", genAiSystem));
  agentAttributes.push(buildAttr("gh-aw.engine", engineId));
}

Meanwhile, the setup‐span attribute build (send_otlp_span.cjs:985-987) and the conclusion‐span attribute build (send_otlp_span.cjs:1422) only emit gh-aw.engine.id:

// setup span – send_otlp_span.cjs:985-987
if (engineId) {
  attributes.push(buildAttr("gh-aw.engine.id", engineId));
}

// conclusion span – send_otlp_span.cjs:1422
if (engineId) attributes.push(buildAttr("gh-aw.engine.id", engineId));
Proposed Change

Extract a small helper next to ENGINE_TO_SYSTEM_MAP that returns the GenAI attribute pair, then call it at all three sites:

// actions/setup/js/send_otlp_span.cjs (near ENGINE_TO_SYSTEM_MAP, ~line 44)

/**
 * Build the OTel GenAI semantic-convention attributes for an engine ID.
 * Returns an empty array when engineId is falsy so callers can spread
 * unconditionally.
 *
 * `@param` {string} engineId
 * `@returns` {Array<{key: string, value: object}>}
 */
function buildEngineAttributes(engineId) {
  if (!engineId) return [];
  const genAiSystem = ENGINE_TO_SYSTEM_MAP[engineId] || engineId;
  return [
    buildAttr("gh-aw.engine.id", engineId),
    buildAttr("gh-aw.engine", engineId),
    buildAttr("gen_ai.system", genAiSystem),
  ];
}

Then replace the three engine-emission sites:

// setup span – replace lines 985-987
attributes.push(...buildEngineAttributes(engineId));

// conclusion span – replace line 1422 (also drop the duplicate emission below)
attributes.push(...buildEngineAttributes(engineId));

// dedicated agent span – replace lines 1648-1652
agentAttributes.push(...buildEngineAttributes(engineId));

(The existing agent‐span block already pushes the same three keys; the helper consolidates them and removes duplication.)

Expected Outcome

After this change:

  • In Grafana / Honeycomb / Datadog: native GenAI dashboards light up for every gh‐aw span that knows the engine — not just the inner LLM call. Filters like gen_ai.system = "anthropic" now return setup, agent, and conclusion spans in one view.
  • In the JSONL mirror (/tmp/gh-aw/otel.jsonl): every span that includes gh-aw.engine.id will also include gen_ai.system, making local jq triage by provider trivial:
    jq -c '.resourceSpans[]?.scopeSpans[]?.spans[]? | select(any(.attributes[]?; .key == "gen_ai.system" and .value.stringValue == "anthropic"))' /tmp/gh-aw/otel.jsonl
  • For on‐call engineers: a single filter (gen_ai.system = anthropic) returns the full lifecycle of every Claude run, including run‐level success/failure status — no more pivoting through gh-aw.engine.id to join setup and conclusion spans manually.
  • Zero risk of double‐counting tokens: gen_ai.usage.* attributes remain exclusively on the dedicated agent span (the change touches only identification, not measurement).
Implementation Steps
  • Add buildEngineAttributes(engineId) helper in actions/setup/js/send_otlp_span.cjs next to ENGINE_TO_SYSTEM_MAP.
  • Replace the three call sites (send_otlp_span.cjs:985-987, :1422, :1648-1652) with the helper.
  • Update actions/setup/js/send_otlp_span.test.cjs, action_setup_otlp.test.cjs, and action_conclusion_otlp.test.cjs to assert that gen_ai.system is present on setup and conclusion spans whenever gh-aw.engine.id is present.
  • Add (or extend) a test that exercises each engine in ENGINE_TO_SYSTEM_MAP to lock the mapping.
  • Run make test-unit (or cd actions/setup/js && npx vitest run) and confirm green.
  • Run make fmt.
  • Open a PR referencing this issue.
Evidence from Live Telemetry

Sentry MCP returned an empty tool list in this run, so the analysis used the local OTLP JSONL mirror (the cheaper trustworthy source per skills/otel-queries/SKILL.md).

  • Source: /tmp/gh-aw/otel.jsonl for the current workflow run (workflow run §25911916631).
  • Span observed: name="gh-aw.agent.setup", traceId="fbe07c7825a564693296f046ee937a09", kind=1, status.code=1, 12 span attributes.
  • Span attribute keys present: gh-aw.job.name, gh-aw.workflow.name, gh-aw.run.id, gh-aw.run.attempt, gh-aw.run.actor, gh-aw.repository, gh-aw.event_name, gh-aw.staged, gh-aw.episode.id, gh-aw.episode.kind, gh-aw.hop.id, gh-aw.workflow_call.id.
  • Span attribute keys absent (but expected for a Claude workflow): gh-aw.engine.id, gh-aw.engine, gen_ai.system. Resource attributes themselves are exemplary (18 keys including service.version, deployment.environment, github.repository, github.run_id, github.event_name, github.workflow_ref, runner.*) — the gap is strictly on span attributes, not resource attributes.
  • Why the engine attributes are absent on this setup span: GH_AW_INFO_ENGINE_ID is injected only on the later generate_aw_info step (e.g. .github/workflows/daily-otel-instrumentation-advisor.lock.yml:131), which runs after ./actions/setup (line 118). The setup step therefore never sees the engine. This means even after the proposed helper is added, the setup span will still emit gen_ai.system only when the engine is resolvable — which today is the case for jobs downstream of the activation job (where aw_info.json is hydrated from artifact). The fix delivers immediate value on conclusion and agent spans, and prepares the ground for a separate, smaller follow‐up that promotes GH_AW_INFO_ENGINE_ID to a job‐level env var so the setup span is also covered.
Related Files
  • actions/setup/js/send_otlp_span.cjs — primary edit site.
  • actions/setup/js/send_otlp_span.test.cjs — assertion updates.
  • actions/setup/js/action_setup_otlp.test.cjs — assert gen_ai.system on setup span when engine is resolvable.
  • actions/setup/js/action_conclusion_otlp.test.cjs — assert gen_ai.system on conclusion span.
  • actions/setup/js/action_otlp.test.cjs — end‐to‐end attribute regression.

Generated by the Daily OTel Instrumentation Advisor workflow

Generated by 📊 Daily OTel Instrumentation Advisor · ● 10.5M · ◷

  • expires on May 22, 2026, 10:08 AM UTC

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions