Repository navigation
Description
📡 OTel Instrumentation Improvement: emit gen_ai.system on setup and conclusion spans
Analysis Date: 2026-05-15
Priority: Medium
Effort: Small (< 2h)
Problem
The OTel GenAI semantic-convention attribute gen_ai.system is currently emitted only on the dedicated gh-aw.<job>.agent span (actions/setup/js/send_otlp_span.cjs:1648–1652). The gh-aw.<job>.setup and gh-aw.<job>.conclusion spans carry gh-aw.engine.id (when resolvable) but never gen_ai.system, even though the engine→system mapping is already defined in ENGINE_TO_SYSTEM_MAP at the top of the same file (send_otlp_span.cjs:38–43).
As a result, a DevOps engineer cannot answer questions such as:
- “What is the end‐to‐end run failure rate per LLM system (
anthropicvsopenaivsgithub_models)?” - “Which
gen_ai.systemhas the worst setup→conclusion p95 wall‐clock latency?” - “Show me all spans in the OTel‐native GenAI dashboard for our
anthropicworkflows.”
Today these queries only return the inner agent span — they silently miss the job lifecycle around it.
Why This Matters (DevOps Perspective)
Grafana, Honeycomb, Datadog, and Sentry all expose built‐in GenAI dashboards that auto‐detect spans by the gen_ai.system attribute (per the OpenTelemetry GenAI semantic conventions). Because gh‐aw only stamps gen_ai.system on the innermost LLM call span, those dashboards present only LLM‐request latency and miss:
- Setup‐phase regressions (e.g. MCP startup time) attributable to a particular engine.
- End‐to‐end run outcomes (success/failure/timeout) grouped by engine.
- Cost / latency joins between
gh-aw.action_minutes(on the conclusion span) and the engine.
In an incident (“claude runs are slow today”), the on‐call engineer must currently pivot through gh-aw.engine.id — a gh‐aw‐specific attribute that few backends understand out‐of‐the‐box — and re‐join setup and conclusion spans manually. With gen_ai.system on every lifecycle span, the same question resolves in one filter.
The mapping table that makes this trivial already exists; we just are not applying it everywhere the engine is known.
Current Behavior
The engine→system mapping is defined once:
// actions/setup/js/send_otlp_span.cjs:38-43
const ENGINE_TO_SYSTEM_MAP = Object.assign(Object.create(null), {
copilot: "github_models",
claude: "anthropic",
codex: "openai",
gemini: "google_vertex_ai",
});But it is consumed only inside the dedicated agent span block:
// actions/setup/js/send_otlp_span.cjs:1648-1652 (only place gen_ai.system is emitted)
if (engineId) {
const genAiSystem = ENGINE_TO_SYSTEM_MAP[engineId] || engineId;
agentAttributes.push(buildAttr("gen_ai.system", genAiSystem));
agentAttributes.push(buildAttr("gh-aw.engine", engineId));
}Meanwhile, the setup‐span attribute build (send_otlp_span.cjs:985-987) and the conclusion‐span attribute build (send_otlp_span.cjs:1422) only emit gh-aw.engine.id:
// setup span – send_otlp_span.cjs:985-987
if (engineId) {
attributes.push(buildAttr("gh-aw.engine.id", engineId));
}
// conclusion span – send_otlp_span.cjs:1422
if (engineId) attributes.push(buildAttr("gh-aw.engine.id", engineId));Proposed Change
Extract a small helper next to ENGINE_TO_SYSTEM_MAP that returns the GenAI attribute pair, then call it at all three sites:
// actions/setup/js/send_otlp_span.cjs (near ENGINE_TO_SYSTEM_MAP, ~line 44)
/**
* Build the OTel GenAI semantic-convention attributes for an engine ID.
* Returns an empty array when engineId is falsy so callers can spread
* unconditionally.
*
* `@param` {string} engineId
* `@returns` {Array<{key: string, value: object}>}
*/
function buildEngineAttributes(engineId) {
if (!engineId) return [];
const genAiSystem = ENGINE_TO_SYSTEM_MAP[engineId] || engineId;
return [
buildAttr("gh-aw.engine.id", engineId),
buildAttr("gh-aw.engine", engineId),
buildAttr("gen_ai.system", genAiSystem),
];
}Then replace the three engine-emission sites:
// setup span – replace lines 985-987
attributes.push(...buildEngineAttributes(engineId));
// conclusion span – replace line 1422 (also drop the duplicate emission below)
attributes.push(...buildEngineAttributes(engineId));
// dedicated agent span – replace lines 1648-1652
agentAttributes.push(...buildEngineAttributes(engineId));(The existing agent‐span block already pushes the same three keys; the helper consolidates them and removes duplication.)
Expected Outcome
After this change:
- In Grafana / Honeycomb / Datadog: native GenAI dashboards light up for every gh‐aw span that knows the engine — not just the inner LLM call. Filters like
gen_ai.system = "anthropic"now return setup, agent, and conclusion spans in one view. - In the JSONL mirror (
/tmp/gh-aw/otel.jsonl): every span that includesgh-aw.engine.idwill also includegen_ai.system, making localjqtriage by provider trivial:jq -c '.resourceSpans[]?.scopeSpans[]?.spans[]? | select(any(.attributes[]?; .key == "gen_ai.system" and .value.stringValue == "anthropic"))' /tmp/gh-aw/otel.jsonl - For on‐call engineers: a single filter (
gen_ai.system = anthropic) returns the full lifecycle of every Claude run, including run‐level success/failure status — no more pivoting throughgh-aw.engine.idto join setup and conclusion spans manually. - Zero risk of double‐counting tokens:
gen_ai.usage.*attributes remain exclusively on the dedicated agent span (the change touches only identification, not measurement).
Implementation Steps
- Add
buildEngineAttributes(engineId)helper inactions/setup/js/send_otlp_span.cjsnext toENGINE_TO_SYSTEM_MAP. - Replace the three call sites (
send_otlp_span.cjs:985-987,:1422,:1648-1652) with the helper. - Update
actions/setup/js/send_otlp_span.test.cjs,action_setup_otlp.test.cjs, andaction_conclusion_otlp.test.cjsto assert thatgen_ai.systemis present on setup and conclusion spans whenevergh-aw.engine.idis present. - Add (or extend) a test that exercises each engine in
ENGINE_TO_SYSTEM_MAPto lock the mapping. - Run
make test-unit(orcd actions/setup/js && npx vitest run) and confirm green. - Run
make fmt. - Open a PR referencing this issue.
Evidence from Live Telemetry
Sentry MCP returned an empty tool list in this run, so the analysis used the local OTLP JSONL mirror (the cheaper trustworthy source per skills/otel-queries/SKILL.md).
- Source:
/tmp/gh-aw/otel.jsonlfor the current workflow run (workflow run §25911916631). - Span observed:
name="gh-aw.agent.setup",traceId="fbe07c7825a564693296f046ee937a09",kind=1,status.code=1, 12 span attributes. - Span attribute keys present:
gh-aw.job.name,gh-aw.workflow.name,gh-aw.run.id,gh-aw.run.attempt,gh-aw.run.actor,gh-aw.repository,gh-aw.event_name,gh-aw.staged,gh-aw.episode.id,gh-aw.episode.kind,gh-aw.hop.id,gh-aw.workflow_call.id. - Span attribute keys absent (but expected for a Claude workflow):
gh-aw.engine.id,gh-aw.engine,gen_ai.system. Resource attributes themselves are exemplary (18 keys includingservice.version,deployment.environment,github.repository,github.run_id,github.event_name,github.workflow_ref,runner.*) — the gap is strictly on span attributes, not resource attributes. - Why the engine attributes are absent on this setup span:
GH_AW_INFO_ENGINE_IDis injected only on the latergenerate_aw_infostep (e.g..github/workflows/daily-otel-instrumentation-advisor.lock.yml:131), which runs after./actions/setup(line 118). The setup step therefore never sees the engine. This means even after the proposed helper is added, the setup span will still emitgen_ai.systemonly when the engine is resolvable — which today is the case for jobs downstream of the activation job (whereaw_info.jsonis hydrated from artifact). The fix delivers immediate value on conclusion and agent spans, and prepares the ground for a separate, smaller follow‐up that promotesGH_AW_INFO_ENGINE_IDto a job‐level env var so the setup span is also covered.
Related Files
actions/setup/js/send_otlp_span.cjs— primary edit site.actions/setup/js/send_otlp_span.test.cjs— assertion updates.actions/setup/js/action_setup_otlp.test.cjs— assertgen_ai.systemon setup span when engine is resolvable.actions/setup/js/action_conclusion_otlp.test.cjs— assertgen_ai.systemon conclusion span.actions/setup/js/action_otlp.test.cjs— end‐to‐end attribute regression.
Generated by the Daily OTel Instrumentation Advisor workflow
Generated by 📊 Daily OTel Instrumentation Advisor · ● 10.5M · ◷
- expires on May 22, 2026, 10:08 AM UTC
Activity