bug: agent server fails to start on every OTEL-enabled agent — startup.sh's unpinned per-boot pip upgrade pulls FastAPI 0.142.0, whose auto-telemetry refuses grpc #3106

Summary

Every agent container with OpenTelemetry enabled fails to start its agent server on its next boot. The cause is the base image's startup.sh, which upgrades FastAPI unpinned on every container start, and so now installs FastAPI 0.142.0. That release adds automatic OpenTelemetry export, configured from the standard OTEL_* environment at ASGI lifespan startup, and it refuses the grpc protocol Trinity injects into every OTEL-enabled agent. The container stays Up because startup.sh keeps it alive, but nothing listens on :8000. Chat, schedules, channels and agent-to-agent calls then fail with transport circuit breaker open / ConnectError.

It is not tied to any Trinity commit. It fires on the next restart or recreate of an affected agent (a platform /update rebuild pass, a host reboot under unless-stopped, a drift recreate on start, a manual restart), on any Trinity version. Rolling back the base image does not help, because the per-boot upgrade runs either way. On one production instance, 29 of 38 agents went down at once during a routine agent rebuild. The 9 that survived were exactly the agents created with no OTEL_* env.

Component

Agent Runtime (base image startup.sh + agent server), with the OTEL env wiring in the backend.

Priority

P0: service down for every OTEL-enabled agent (OTEL_ENABLED defaults to 1) on its next boot; no warning, containers report healthy-looking Up.

Error

With the default OTEL_EXPORTER_OTLP_PROTOCOL=grpc:

INFO:     Waiting for application startup.
ERROR:    FastAPI automatic telemetry requires the OTLP http/protobuf protocol. Configure other transports explicitly and pass telemetry={'auto_configure': False} to FastAPI().
ERROR:    Application startup failed. Exiting.

Switching the platform to OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf (+ the collector's :4318) does not help. The next check fails instead, because the extra is not installed:

ERROR:    Automatic OpenTelemetry export requires fastapi[opentelemetry] or fastapi[standard]. Install the extra, configure providers yourself, or pass telemetry={'auto_configure': False} to FastAPI().
ERROR:    Application startup failed. Exiting.

Location

• File: docker/base-image/startup.sh
• Line: 611–618 ("Verifying agent-server dependencies…")
• Code: python3 -m pip install --user --quiet --upgrade fastapi uvicorn httpx pydantic python-multipart pyyaml

Related:

• docker/base-image/agent_server/main.py:57: app = FastAPI( constructed with default telemetry (auto-configure on)
• docker/base-image/Dockerfile:218: the same packages are installed unpinned at build time (the image bakes FastAPI 0.141.1; the boot then upgrades to 0.142.0 in ~/.local)
• src/backend/services/agent_service/crud.py:1849, lifecycle.py:1673, system_agent_service.py:419: inject OTEL_EXPORTER_OTLP_PROTOCOL=grpc and OTEL_EXPORTER_OTLP_ENDPOINT=http://trinity-otel-collector:4317 into agents

Root Cause

1. startup.sh runs pip install --upgrade with no version pins on every boot, so each container start silently takes whatever PyPI published most recently. The comment says it exists "to prevent template repos from breaking the agent server", but the same upgrade also accepts a breaking upstream release.
2. FastAPI 0.142.0 fastapi/telemetry/_runtime.py::_export_endpoint raises unless OTEL_EXPORTER_OTLP_PROTOCOL is http/protobuf. Its exporters come from the fastapi[opentelemetry] extra, which the agent image does not install. The only environment escape hatch is OTEL_SDK_DISABLED=true (checked in _configure_from_environment), which Trinity does not set.
3. The agent env's OTEL_* variables are meant for Claude Code (cost/usage metrics to the collector), but FastAPI reads the same standard names and applies them to the agent server itself.

Reproduction Steps

1. Run any Trinity instance with OTEL_ENABLED=1 (the default), with at least one agent created while OTEL was enabled.
2. Restart or recreate that agent (e.g. docker restart agent-<name>, a Stop/Start from the UI, or a base-image rebuild pass).
3. The boot runs the startup.sh upgrade and installs FastAPI ≥ 0.142.0 (docker exec agent-<name> python3 -c 'import fastapi;print(fastapi.__version__)').
4. docker exec agent-<name> curl -s localhost:8000/health → connection refused; docker logs agent-<name> shows the error above.
5. Agents created with OTEL_ENABLED=0 (no OTEL_* env) boot normally.

Suggested Fix

Any one of these fixes the outage; (a) and (b) together are the durable fix.

(a) Opt the agent server out of FastAPI's auto-telemetry. The OTEL_* env belongs to Claude Code, not the agent server:

# docker/base-image/agent_server/main.py
app = FastAPI(
    ...,
    telemetry={"auto_configure": False},  # OTEL_* env is Claude Code's, not the agent server's
)

(guard with a version check / try if older FastAPI must still be supported, since the kwarg is new in 0.142)

(b) Stop taking unpinned upgrades at boot. Pin the agent-server dependencies (a constraints file baked into the image, pip install -c /app/constraints.txt), or drop --upgrade so the boot only repairs missing or broken packages. Pinning build time as well (Dockerfile:218) makes image rebuilds reproducible too, the same lesson as the CLAUDE_CODE_VERSION pin in #3012.

(c) Interim, platform-side: inject OTEL_SDK_DISABLED=true alongside the other OTEL_* vars in crud.py, lifecycle.py (recovery) and system_agent_service.py. Verified in production: FastAPI skips auto-configuration, and the Claude Code binary contains no reference to OTEL_SDK_DISABLED (it reads OTEL_METRICS_EXPORTER, OTEL_EXPORTER_OTLP_PROTOCOL, CLAUDE_CODE_ENABLE_TELEMETRY), so its grpc metrics export keeps working. Metrics reached the collector normally after the change.

Operator workaround today (no code change): recreate each affected agent via recreate_container_with_updated_config(name, old, owner, env_overrides={'OTEL_SDK_DISABLED': 'true'}, require_running=True, preserve_run_state=True). The override survives later recreates (Config.Env replay) but not the DB-based recovery path, and new agents do not get it.

A regression test that boots the agent server with OTEL_EXPORTER_OTLP_PROTOCOL=grpc + an endpoint set, against the latest allowed FastAPI, would have caught this.

Environment

• Trinity version: dev @ 9690bc89 (not version-specific: triggered by the boot-time upgrade)
• FastAPI: 0.141.1 baked in the image → 0.142.0 installed at boot
• Claude Code: 2.1.281 (agent base image)
• Docker version: 28.3.3
• OS: Debian GNU/Linux 12 (bookworm) host

Related

• docker/base-image/startup.sh
• docker/base-image/agent_server/main.py
• docker/base-image/Dockerfile
• Default model claude-opus-5-5 fails on cached Claude Code 2.1.278, and the rejection is misclassified as auth (seat rotation + API-key fallback) #3012 (same class: an unpinned @latest in the base image)