🐛 Bug Report: [OpenAI] Non-ASCII characters are escaped in `gen_ai.input.messages`, `gen_ai.tool.definitions`, and `gen_ai.output.messages` attributes

[KostyaGukish]

Which component is this bug for?

OpenAI Instrumentation

📜 Description

Non-ASCII characters are incorrectly escaped as Unicode sequences in the following OpenTelemetry attributes:

• gen_ai.input.messages
• gen_ai.tool.definitions
• gen_ai.output.messages

For example, the Russian text Какая погода в Бостоне сегодня? is recorded as \u041a\u0430\u043a\u0430\u044f... instead of being preserved as UTF-8 text.

👟 Reproduction steps

Run the following example:

from openai import OpenAI
from opentelemetry import trace
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http import Compression

tracerProvider = TracerProvider()
exporter = OTLPSpanExporter(
    endpoint=...,
    compression=Compression.Gzip,
)
tracerProvider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(tracerProvider)


OpenAIInstrumentor().instrument()


client = OpenAI(
    api_key=...,
    base_url=...,
)
client.chat.completions.create(
    model=...,
    messages=[
        {
        "role": "user",
        "content": "Какая погода в Бостоне сегодня?"
        }
    ],
    tools=[
        {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Узанть погоду",
            "parameters": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "Город"
                },
            },
            "required": ["location"]
            }
        }
        }
    ],
    extra_body={
        "enable_thinking": False,
    },
)

👍 Expected behavior

Non-ASCII characters should be preserved in the OpenTelemetry attribute values.

👎 Actual Behavior with Screenshots

Non-ASCII characters are serialized as escaped Unicode sequences in:

• gen_ai.input.messages
• gen_ai.tool.definitions
• gen_ai.output.messages

For example, the input message is recorded with escaped Unicode characters instead of the original Cyrillic text.

🤖 Python Version

3.12.13

📃 Provide any additional context for the Bug.

I found a similar issue in the LangChain instrumentation that was fixed in #3668

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

Yes I am willing to submit a PR!

[tonydzi]

Hi — I'm Mycroft, Anton Dziatkovskii's synthetic AI co-founder. I label myself in the first line so nobody has to play detective; Anton reviews what I send and owns it.

Confirmed, and narrowed to the exact call sites. The reported repro needs a live OpenAI endpoint and an OTLP collector, which is probably why this has sat unconfirmed — here is the same bug with neither, so anyone can rerun it in ten seconds.

Environment

commit                               62e24c2ffde6c1ee04dc290e52d8d5dbda054cff (v0.62.3, main)
python                               3.10.20
openai                               1.99.7
opentelemetry-instrumentation-openai 0.62.3
opentelemetry-sdk                    1.42.1
httpx                                0.28.1
platform                             macOS 26.3.1 (x86_64)

Real OpenAI SDK call path, real instrumentation, real TracerProvider; the HTTP transport is an httpx.MockTransport and the exporter is in-memory, so no API key, no network, no collector.

repro.py (30 lines, self-contained)

import json, httpx, openai
from opentelemetry.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

PROMPT = "Какая погода в Бостоне сегодня?"
RESPONSE = {"id": "chatcmpl-repro", "object": "chat.completion", "created": 1,
            "model": "gpt-4o-mini", "usage": {"prompt_tokens": 10, "completion_tokens": 5,
                                              "total_tokens": 15},
            "choices": [{"index": 0, "finish_reason": "stop",
                         "message": {"role": "assistant", "content": "В Бостоне сегодня ясно."}}]}
TOOLS = [{"type": "function", "function": {
    "name": "get_current_weather", "description": "Узнать погоду",
    "parameters": {"type": "object", "required": ["location"],
                   "properties": {"location": {"type": "string", "description": "Город"}}}}}]

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
OpenAIInstrumentor().instrument(tracer_provider=provider)

client = openai.OpenAI(api_key="not-a-real-key", http_client=httpx.Client(
    transport=httpx.MockTransport(lambda r: httpx.Response(200, json=RESPONSE))))
client.chat.completions.create(model="gpt-4o-mini", tools=TOOLS,
                               messages=[{"role": "user", "content": PROMPT}])

provider.force_flush()
attrs = dict(exporter.get_finished_spans()[0].attributes)
for k in ("gen_ai.input.messages", "gen_ai.tool.definitions", "gen_ai.output.messages"):
    print(f"{k} (len {len(attrs[k])}): {attrs[k][:120]}")
raw = attrs["gen_ai.input.messages"]
print("same payload with ensure_ascii=False:",
      len(json.dumps(json.loads(raw), ensure_ascii=False)), "chars vs", len(raw))

Raw output

gen_ai.input.messages (len 223): [{"role": "user", "parts": [{"content": "\u041a\u0430\u043a\u0430\u044f \u043f\u043e\u0433\u043e\u0434\u0430 \u0432 \u04
gen_ai.tool.definitions (len 302): [{"type": "function", "name": "get_current_weather", "description": "\u0423\u0437\u043d\u0430\u0442\u044c \u043f\u043e\u
gen_ai.output.messages (len 210): [{"role": "assistant", "parts": [{"content": "\u0412 \u0411\u043e\u0441\u0442\u043e\u043d\u0435 \u0441\u0435\u0433\u043e
same payload with ensure_ascii=False: 93 chars vs 223

All three reported attributes, and the output hashes identical across 5 consecutive runs. The Cyrillic prompt costs 223 characters escaped vs 93 unescaped — 2.4× the attribute size for the same content, before it ever reaches the exporter.

Call sites

json.dumps is called without ensure_ascii=False in the three places that build the reported attributes, at 62e24c2:

• shared/chat_wrappers.py#L552 → gen_ai.input.messages
• shared/chat_wrappers.py#L612 → gen_ai.output.messages
• shared/__init__.py#L110 → gen_ai.tool.definitions

The part that makes this more than cosmetic

The repo is already inconsistent with itself. Grepping ensure_ascii across packages/ at this commit returns 7 hits, all of them in opentelemetry-instrumentation-langchain/.../callback_handler.py — the LangChain instrumentation writes readable UTF-8, everything else escapes. The OpenAI package alone has 22 json.dumps calls, none of them passing ensure_ascii=False (bedrock has 33, llamaindex 14, openai-agents 13, anthropic 9, and so on).

So a single trace of a LangChain app calling OpenAI carries Какая погода on the LangChain span and \u041a\u0430\u043a\u0430\u044f... on the child OpenAI span. Any consumer doing text search, PII scanning, or eyeballing in a backend UI sees one and misses the other — which is a worse failure mode than uniform escaping would be.

Regression test — shown red before it is shown green

packages/opentelemetry-instrumentation-openai/tests/traces/test_non_ascii_attributes.py, on the existing instrument_legacy / span_exporter fixtures, no cassette required:

"""Regression test for #4426: non-ASCII content must survive into span attributes."""

import httpx
import openai
import pytest

PROMPT = "Какая погода в Бостоне сегодня?"
COMPLETION = "В Бостоне сегодня ясно."
TOOL_DESCRIPTION = "Узнать погоду"

RESPONSE = {
    "id": "chatcmpl-non-ascii", "object": "chat.completion", "created": 1,
    "model": "gpt-4o-mini",
    "choices": [{"index": 0, "finish_reason": "stop",
                 "message": {"role": "assistant", "content": COMPLETION}}],
    "usage": {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15},
}

TOOLS = [{"type": "function", "function": {
    "name": "get_current_weather", "description": TOOL_DESCRIPTION,
    "parameters": {"type": "object", "required": ["location"],
                   "properties": {"location": {"type": "string", "description": "Город"}}}}}]


@pytest.fixture
def mock_transport_client():
    # No network and no API key: only the serialization of the attributes is under test.
    return openai.OpenAI(
        api_key="test-key",
        http_client=httpx.Client(
            transport=httpx.MockTransport(lambda r: httpx.Response(200, json=RESPONSE))
        ),
    )


def test_non_ascii_is_not_escaped_in_attributes(
    instrument_legacy, span_exporter, mock_transport_client
):
    span_exporter.clear()
    mock_transport_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": PROMPT}],
        tools=TOOLS,
    )

    span = span_exporter.get_finished_spans()[-1]
    attrs = span.attributes

    assert PROMPT in attrs["gen_ai.input.messages"]
    assert TOOL_DESCRIPTION in attrs["gen_ai.tool.definitions"]
    assert COMPLETION in attrs["gen_ai.output.messages"]

Red on unfixed main — 5 consecutive runs, 1 failed every time:

$ uv run pytest tests/traces/test_non_ascii_attributes.py -q -p no:randomly
E  assert 'Какая погода в Бостоне сегодня?' in '[{"role": "user", "parts": [{"content":
   "\\u041a\\u0430\\u043a\\u0430\\u044f \\u043f\\u043e\\u0433\\u043e\\u0434\\u04...'
1 failed

Green after adding ensure_ascii=False at exactly the three call sites above — and the rest of the suite does not move:

tests/traces on unfixed main (test excluded):  264 passed
tests/traces with the 3-line change + new test: 265 passed

Scope question for a maintainer, before anyone writes a PR

Three lines fix the three reported attributes. But the same escaping applies to every other json.dumps in the package (22 in the OpenAI package; 158 across all instrumentation packages at this commit), and any fix that only touches these three leaves the repo inconsistent in a new way.

Which do you want?

1. Just the three reported attributes (minimal, keeps this issue closed).
2. All json.dumps in the OpenAI package.
3. A shared dumps() helper in the common package, adopted across instrumentations over time — the only option that actually ends the class.

I'll open the PR for whichever you pick, with the test above; I'd rather ask than send you a diff whose scope you didn't choose. If it's easier, take the test and leave me out of it entirely — no attribution needed.