Align telemetry with OpenTelemetry MCP semantic conventions

Summary

OpenTelemetry officially merged MCP semantic conventions on January 12, 2026 (PR #2083). ToolHive should align its telemetry implementation with these standards for better observability tool compatibility and ecosystem alignment.

Standard References

Main Documentation: docs/gen-ai/mcp.md
Attribute Registry: model/mcp/registry.yaml
Metrics Definitions: model/mcp/metrics.yaml
W3C Trace Context: https://www.w3.org/TR/trace-context/

Current State

ToolHive has solid telemetry foundation but predates the official conventions:

Middleware: pkg/telemetry/middleware.go - spans, attributes, metrics for MCP proxy
vMCP: pkg/vmcp/server/telemetry.go - backend and workflow telemetry
Parser: pkg/mcp/parser.go - already extracts _meta field (lines 228-233)

Core Implementation Tasks

1. Update Attributes and Span Naming

File: pkg/telemetry/middleware.go

Attribute Renames (for standard compliance):

mcp.method → mcp.method.name (line 222)
mcp.request.id → jsonrpc.request.id (line 229)
mcp.tool.name → gen_ai.tool.name (line 263)
mcp.tool.arguments → gen_ai.tool.call.arguments (line 267, opt-in)
mcp.prompt.name → gen_ai.prompt.name (line 279)
mcp.transport → network.transport with value mapping:
- stdio → pipe
- sse, streamable-http → tcp

Add Missing Required Attributes:

mcp.protocol.version - MCP spec version (e.g., "2025-11-25")
mcp.session.id - Session identifier
jsonrpc.protocol.version - When not "2.0"
error.type - On failures (JSON-RPC error code or "tool_error")
rpc.response.status_code - When response contains error
gen_ai.operation.name - "execute_tool" for tool calls
network.protocol.name - "http" for SSE/streamable-http

Span Naming (lines 161-170):

Current: mcp.tools/call
Standard: tools/call get_weather (include target when available)
Format: {mcp.method.name} {target} where target is tool/prompt name

2. Add Standard Metrics

File: pkg/telemetry/middleware.go

New Standard Metrics (alongside existing):

mcp.client.operation.duration (histogram, seconds)
mcp.server.operation.duration (histogram, seconds)
Use recommended buckets: [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60, 120, 300]

Keep existing toolhive_mcp_* metrics for backward compatibility.

3. Implement W3C Trace Context Propagation

Critical Feature: Enable distributed tracing across MCP boundaries.

3a. Context Injection (vMCP → Backends)

Files:

New: pkg/telemetry/propagation.go - W3C Trace Context helpers
pkg/vmcp/client/client.go - Inject before backend calls

Implementation: Inject traceparent and tracestate into params._meta:

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "get-weather",
    "_meta": {
      "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
      "tracestate": "rojo=00f067aa0ba902b7"
    }
  }
}

Code Structure:

// propagation.go
func InjectTraceContext(ctx context.Context, params map[string]interface{}) {
    meta := getOrCreateMeta(params)
    carrier := &MetaCarrier{meta: meta}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
}

type MetaCarrier struct {
    meta map[string]interface{}
}
// Implement TextMapCarrier interface

3b. Context Extraction (Clients → ToolHive)

File: pkg/telemetry/middleware.go (around line 114)

Implementation: Extract trace context from incoming params._meta and use as parent for server span:

if parsedMCP := mcpparser.GetParsedMCPRequest(ctx); parsedMCP != nil && parsedMCP.Meta != nil {
    carrier := &MetaCarrier{meta: parsedMCP.Meta}
    ctx = otel.GetTextMapPropagator().Extract(ctx, carrier)
}

4. Add Client-Side Spans for vMCP

File: pkg/vmcp/client/client.go

Current: Only SERVER spans when serving requests
Needed: CLIENT spans when vMCP calls backend MCP servers

Operations to Instrument:

initialize - Protocol handshake
tools/list, tools/call
resources/list, resources/read
prompts/list, prompts/get

Span Kind: Use trace.SpanKindClient for these operations.

5. Add Session Duration Metrics

Files:

pkg/vmcp/server/session_adapter.go - Track session lifecycle
Proxy components - Track session termination

Metrics:

mcp.client.session.duration (histogram, seconds)
mcp.server.session.duration (histogram, seconds)

Attributes:

mcp.protocol.version
network.protocol.name
network.transport
error.type (if session terminated with error)

Backward Compatibility

Approach: Emit both legacy and standard names during transition period.

Configuration: Add optional flag:

telemetry:
  useLegacyAttributes: false  # default: standard only

CLI Flag: --otel-use-legacy-attributes (enables dual emission)

Timeline:

Ship standard-compliant attributes/metrics immediately
Announce deprecation after 6 months
Remove legacy support in v2.0

Components Affected

pkg/telemetry/middleware.go - MCP proxy telemetry (spans, metrics, attributes)
pkg/telemetry/propagation.go - New file for trace context helpers
pkg/vmcp/client/client.go - CLIENT spans and context injection
pkg/vmcp/server/session_adapter.go - Session duration tracking
pkg/telemetry/config.go - Backward compatibility configuration
cmd/thv-operator/api/v1alpha1/*_types.go - CRD telemetry specs
docs/observability.md - Update documentation
Test files: Update assertions for new attribute names

Testing Requirements

Update test expectations in pkg/telemetry/middleware_test.go
Update E2E tests in test/e2e/telemetry_middleware_e2e_test.go
Add trace propagation E2E test (vMCP → backend → vMCP chain)
Validate span hierarchy (CLIENT/SERVER relationship)
Test session duration tracking
Verify histogram buckets

Success Criteria

References

OTel MCP PR: MCP semantic conventions open-telemetry/semantic-conventions#2083
Implementation Examples:
- .NET: OpenTelemetry: context propagation and semconv update modelcontextprotocol/csharp-sdk#262
- Python: https://github.com/mayankagarwals/openllmetry/tree/main/packages/opentelemetry-instrumentation-mcp
Related Issue: Mentioned in team discussion about enhancing ToolHive telemetry

@ChrisJBurns

@ChrisJBurns Thanks for picking this up! I added "Traces produced by vMCP also include spans from the MCPServer's proxyrunner" as a Success Criteria. I noticed while debugging an issue today that the traces are broken, and this should be fixed by the 3b work.

Align telemetry middleware with MCP OTEL semantic conventions

Add CLIENT spans and trace context injection for vMCP backends

Add comprehensive OTEL telemetry documentation and migration guide

Implementation Summary

All success criteria from this issue have been addressed across 13 merged PRs. Here's how each criterion was met, which PRs contributed, and what remains as future work.

✅ All required attributes emitted per standard

Rename span attributes to MCP OTEL semantic conventions #3748 — Renamed all span attributes to OTEL MCP semantic convention names (mcp.method → mcp.method.name, mcp.request.id → jsonrpc.request.id, mcp.tool.name → gen_ai.tool.name, mcp.transport → network.transport, etc.). Added new attributes with no legacy equivalent (jsonrpc.protocol.version, gen_ai.operation.name, gen_ai.prompt.name, network.protocol.name, url.scheme, url.path, server.address, user_agent.original).
Add network/client span attributes and fix span status per OTEL semconv #3750 — Added network.protocol.version, mcp.backend.protocol.version, client.address, client.port, mcp.session.id, and error.type on 5xx errors. Fixed span status to follow OTEL HTTP server span conventions (4xx → Unset, 5xx → Error).
Add mcp.protocol.version, client operation duration, span target naming, and client span kind #3757 — Added mcp.protocol.version from MCP-Protocol-Version header and SpanKindClient on backend calls.
Align vMCP client telemetry with OTEL MCP semantic conventions #3775 — Added mcp.method.name, gen_ai.tool.name, gen_ai.prompt.name, mcp.resource.uri to vMCP client spans.

✅ Span names follow `{method} {target}` format

Rename span attributes to MCP OTEL semantic conventions #3748 — Updated proxy middleware span naming from mcp.tools/call to tools/call.
Add mcp.protocol.version, client operation duration, span target naming, and client span kind #3757 — Extended span names to include target: tools/call github_search, prompts/get my-prompt.
Align vMCP client telemetry with OTEL MCP semantic conventions #3775 — Aligned vMCP client span names from internal format (telemetryBackendClient.call_tool) to spec format (tools/call my-tool).

✅ Standard metrics recorded with correct units/attributes

Add mcp.server.operation.duration metric per OTEL MCP semantic conventions #3752 — Added mcp.server.operation.duration histogram metric with spec-defined attributes and bucket boundaries ([0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60, 120, 300]).
Add mcp.protocol.version, client operation duration, span target naming, and client span kind #3757 — Added mcp.client.operation.duration histogram metric to the vMCP backend client with correct buckets and attributes.
Align vMCP client telemetry with OTEL MCP semantic conventions #3775 — Added spec-required attributes (mcp.method.name, network.transport, error.type) to the mcp.client.operation.duration metric.

✅ W3C Trace Context propagates through `params._meta`

Add W3C Trace Context propagation via MCP _meta #3682 & Inject W3C Trace Context into vMCP backend HTTP requests #3814— Added MetaCarrier implementing propagation.TextMapCarrier for MCP _meta fields, plus InjectMetaTraceContext for injecting traceparent/tracestate into outgoing MCP requests.
Extract trace context from MCP _meta for distributed tracing #3763 — Wired MetaCarrier into the telemetry middleware to extract traceparent/tracestate from incoming MCP params._meta, completing the distributed tracing chain.

✅ CLIENT spans created for vMCP backend calls

Add mcp.protocol.version, client operation duration, span target naming, and client span kind #3757 — vMCP backend call spans now use trace.SpanKindClient instead of the default Internal kind.
Align vMCP client telemetry with OTEL MCP semantic conventions #3775 — Full alignment of vMCP client spans with OTEL MCP semantic conventions (attributes, span names, metric attributes).

✅ Network transport values mapped correctly (stdio→pipe, http→tcp)

Rename span attributes to MCP OTEL semantic conventions #3748 — Added mapTransport helper to convert MCP transport types to OTEL network.transport values (stdio → pipe, sse/streamable-http → tcp).
Align vMCP client telemetry with OTEL MCP semantic conventions #3775 — Added mapTransportTypeToNetworkTransport for vMCP client-side transport mapping.

✅ Backward compatibility maintained (with flag)

Add UseLegacyAttributes flag to telemetry config #3729 — Added UseLegacyAttributes flag to telemetry config, plumbed through CLI (--otel-use-legacy-attributes), config file, and operator CRD types. Defaults to true.
Gate legacy span attributes behind UseLegacyAttributes flag #3747 — Gated all existing span attribute emissions behind if m.config.UseLegacyAttributes conditionals.
Rename span attributes to MCP OTEL semantic conventions #3748 — New standard attributes always emitted; legacy names conditionally emitted behind the flag.

✅ Documentation updated

Add OTEL telemetry documentation and migration guide #3784 — Updated docs/observability.md with complete metrics reference, span attributes, configuration details, and distributed tracing. Updated docs/operator/virtualmcpserver-observability.md with vMCP metric tables. Added new docs/telemetry-migration-guide.md with legacy→new attribute mapping and step-by-step migration.

✅ Bug fixes discovered during implementation

Fix SSE gauge leak, config bool handling, and metric error logging #3756 — Fixed SSE active connections gauge leak (incremented but never decremented), config use-legacy-attributes: false being silently ignored (Go zero-value issue), and metric creation errors being silently discarded.
Fix tool counter per-request creation, omitempty config, and streamable-http version #3761 — Fixed tool counter per-request creation, omitempty config serialization bug, and incorrect streamable-http protocol version.
Add String() method to telemetry Config to redact OTLP credentials #3782 — Added fmt.Stringer/fmt.GoStringer to telemetry Config to redact OTLP header credentials in logs.

⚠️ Session duration metrics — Deferred

mcp.server.session.duration and mcp.client.session.duration are intentionally deferred pending the session management architecture refactor (RFC THV-0038). Tracked in #3764.

Future Work

The following items are tracked as separate issues for follow-up:

OTEL: Session-related telemetry and code quality issues #3764 — Session-related telemetry and code quality issues (session duration metrics, session lifecycle spans)
OTEL: Add error.type and rpc.response.status_code for JSON-RPC errors #3765 — Add error.type and rpc.response.status_code for JSON-RPC errors
Deprecate redundant custom duration metrics in favor of OTEL MCP standard metrics #3791 — Deprecate redundant custom duration metrics (toolhive_mcp_request_duration, toolhive_vmcp_backend_requests_duration) in favor of the standard mcp.server.operation.duration and mcp.client.operation.duration

Add OTEL telemetry documentation and migration guide (#3784)

Navigation Menu

Repository navigation

Align telemetry with OpenTelemetry MCP semantic conventions #3399

Summary

Standard References

Current State

Core Implementation Tasks

1. Update Attributes and Span Naming

2. Add Standard Metrics

3. Implement W3C Trace Context Propagation

3a. Context Injection (vMCP → Backends)

3b. Context Extraction (Clients → ToolHive)

4. Add Client-Side Spans for vMCP

5. Add Session Duration Metrics

Backward Compatibility

Components Affected

Testing Requirements

Success Criteria

References

Implementation Summary

✅ All required attributes emitted per standard

✅ Span names follow `{method} {target}` format

✅ Standard metrics recorded with correct units/attributes

✅ W3C Trace Context propagates through `params._meta`

✅ CLIENT spans created for vMCP backend calls

✅ Network transport values mapped correctly (stdio→pipe, http→tcp)

✅ Backward compatibility maintained (with flag)

✅ Documentation updated

✅ Bug fixes discovered during implementation

⚠️ Session duration metrics — Deferred

Future Work

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Align telemetry with OpenTelemetry MCP semantic conventions #3399

Description

Summary

Standard References

Current State

Core Implementation Tasks

1. Update Attributes and Span Naming

2. Add Standard Metrics

3. Implement W3C Trace Context Propagation

3a. Context Injection (vMCP → Backends)

3b. Context Extraction (Clients → ToolHive)

4. Add Client-Side Spans for vMCP

5. Add Session Duration Metrics

Backward Compatibility

Components Affected

Testing Requirements

Success Criteria

References

Activity

jerm-dro commented on Feb 5, 2026

ChrisJBurns commented on Feb 12, 2026

Implementation Summary

✅ All required attributes emitted per standard

✅ Span names follow {method} {target} format

✅ Standard metrics recorded with correct units/attributes

✅ W3C Trace Context propagates through params._meta

✅ CLIENT spans created for vMCP backend calls

✅ Network transport values mapped correctly (stdio→pipe, http→tcp)

✅ Backward compatibility maintained (with flag)

✅ Documentation updated

✅ Bug fixes discovered during implementation

⚠️ Session duration metrics — Deferred

Future Work

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

✅ Span names follow `{method} {target}` format

✅ W3C Trace Context propagates through `params._meta`