feat(pod): expose real pod runtime state on pod get and list (CON-690)

[justinwlin]

pod get and pod list only ever showed desiredStatus, so a pod pulling a 20 GB image and a pod serving traffic looked identical, and ssh just said pod not ready with no reason. This adds a derived runtimeStatus + runtimeStatusReason, built only from signals the platform genuinely exposes.

Linear: CON-690

What changed

• New internal/podstate: derives a five-value status (running, initializing, stopped, terminated, unknown) plus reason tokens from desiredStatus, runtime-telemetry presence and lastStatusChange. One derivation, used by pod get, pod list, ssh info and ssh connect.
• pod get reuses the graphql call it already made for the ssh block — no extra round trip.
• pod list makes one bulk myPods call (never N+1), skipped unless some listed pod is RUNNING, capped at 5s, and best-effort: a failure degrades to unknown/runtime_unavailable instead of failing the list. Also carries lastStatusChange now.
• ssh errors name the actual cause, including three distinct reasons for "running but no reachable port 22".
• Two bugs fixed on the way: uptimeSeconds was published as a permanent 0 (the real field is runtime.uptimeInSeconds), and an EXITED pod kept reporting stale runtime ports, so pod get / ssh connect handed back an ssh command that could not connect.
• Additive output contract; docs/ regenerated.

Notable decisions

• No pulling/starting split. PodRuntime exposes no phase field (verified in the backend schema, the host telemetry struct, and live prod probes); initializing covers pull + create + boot as one state. The backend ask is a follow-up.
• No image_pull_error. PodStopReason is never persisted or queryable — the strongest honest token is stopped_by_runpod. Persisting Pod.stopReason is a follow-up.
• runtimeStatus derives from the graphql snapshot, not REST's desiredStatus, so pod get and pod list can never disagree with the ports they display. The two surfaces can still print desiredStatus: RUNNING beside runtimeStatus: stopped during momentary skew — that is the point.
• Not done from the ticket: the machine-fields half of the investigation (maintenance windows would explain a pod stuck initializing). Carried as a follow-up rather than left implicit.

Full deviation analysis and review triage (2 rounds, 29 findings: 22 fixed, 2 rejected, rest disclosed)

Every deviation from the ticket was source-verified by independent reviewers against runpod-backend / host and live prod probes:

ticket asked for	shipped	why
pulling/starting states	one initializing state	no phase field exists anywhere in the pipeline (podRuntime.ts, host/pkg/telemetry/store.go); live probes for runtime.status/.phase/.state all fail validation
image_pull_error (spike brief)	stopped_by_runpod	PodStopReason is input-only; stopPod.ts mails it and drops it — no column, not queryable
"timebox a spike first"	full CLI implementation, no backend PRs	the spike answer was "the field cannot be faked"; everything shipped derives from existing fields
investigate machine fields	not done	disclosed gap; the maintenance fields exist and are a follow-up

Key review fixes across both rounds: pod list and pod get disagreeing under REST/graphql skew (both now derive via sshconnect.PodState); the suggested --ports 22/tcp remediation silently unpublishing every other port (now unions with declared ports and warns the container may restart); pod list stalling 30s on unresponsive graphql (now skipped when nothing is RUNNING, capped at 5s — measured 30.04s → 0.01s empty); user terminates reporting stopped with no reason (now terminated with attribution); Outbid: attribution preserved; stale-ssh guard extended to ssh connect ([], not a dead command).

Rejected: hint placement framing (moved anyway, with the port logic); "report transcripts not verbatim" (report issue, not a code defect).

Known nit, deliberately unchanged (pre-existing shape): ssh info puts an "error" key on stdout with exit 0. Follow-up. (A second nit — remediation strings printing a literal <pod-id> — was fixed in 80a1d19: both call sites interpolate the real id now.)

Testing

internal/podstate 100% coverage; cmd/pod 24.6% → 41.3%. New tests drive pod list, pod get, ssh info and ssh connect end-to-end against a stub REST+graphql control plane. 18 mutations applied across both review rounds — all caught. Live prod e2e observed the initializing → running transition on the cheapest available community GPU ($0.17/hr A4000); all resources deleted, pod list -a empty after.

Evidence: stub transcripts, live e2e, gates

Stub control plane (free):

# running pod declaring only 8888/http -> the suggestion keeps it
$ rp pod get p | jq -r .ssh.error
pod not ready: pod does not publish 22/tcp; add it with 'runpodctl pod update <pod-id>
  --ports 8888/http,22/tcp' (--ports replaces the whole list, and changing it may restart the container)

# rest RUNNING vs graphql EXITED: list and get agree, no stale uptime
$ rp pod list -a   -> runtimeStatus stopped / stopped_by_user
$ rp ssh connect   -> {"connections": []}

# EXITED-only account, graphql sleeping 20s: probe skipped entirely
$ time rp pod list -a    -> real 0.01  (graphql hits: 0)

# running pod, graphql sleeping 20s: 5s cap fires, degrades, exit 0
$ time rp pod list -a    -> real 5.01, runtimeStatus unknown / runtime_unavailable

Live prod (round 1):

[18:45:04] GET  status=initializing reason=awaiting_container uptime=None
[18:45:18] GET  status=running uptime=7
$ pod stop ... && pod get ...   -> runtimeStatus stopped / stopped_by_user, uptimeSeconds absent, no ssh_command
$ ssh connect                   -> {"connections": []}
$ pod delete ... && pod list -a -> []

$ go test -tags e2e -run TestCLI_PodRuntimeStatusTransition ./e2e/...
--- PASS: TestCLI_PodRuntimeStatusTransition (21.52s)

Gates: gofmt -l . clean, go vet ./... + go vet -tags e2e ./e2e/... clean, go test -count=1 --cover ./... all ok, go run ./docs/docs-gen.go no diff. (gofmt was not clean at base; fixed in its own style: commit.)

Follow-ups

• update the runpodctl agent skill — new runtimeStatus/runtimeStatusReason vocabulary, lastStatusChange on pod list, pod update --ports replaces rather than appends
• read Pod.machine maintenance fields (the ticket's unfinished half)
• decide whether ssh info's stdout "error" key should be renamed or documented as an exception
• backend: persist + expose Pod.stopReason; add PodRuntime.phase to split pulling from starting
• pod logs against v2-rest.runpod.io/v2/pods/{id}/logs (already reachable with the current key)
• pre-existing bugs found, untouched: main-ui LayoutLowBalanceBanner case-sensitive match on "Exited by RunPod:"; pod list --output=table silently returns json; FindPodConnection panic on a pods:[null] element (also on main)

[lukepiette]

Approving. Checked the PR body's claims against pre-PR main and live schemas and they hold: pod get adds zero extra round trips (main already called GetPods() for the ssh block), pod list adds exactly one bulk graphql call — never N+1 — capped at 5s and degrading to unknown on failure (all test-pinned), the JSON contract is strictly additive with desiredStatus values untouched, no flags or exit codes changed, runtime.uptimeInSeconds exists in the published schema, and REST really does carry lastStatusChange. No tests deleted or weakened; old behavior is pinned alongside the new.

Comments (none blocking):

1. Release-note the ssh output changes. They are the edges most likely to surface in someone's script: the ssh info error string changed from exactly "pod not ready" to "pod not ready: <detail>" (exact-match consumers break, prefix-match survives); stopped pods no longer get a dead ssh_command and drop out of the connections inventory; empty connections changed from JSON null to []. All correct fixes — the old commands could never connect — but they deserve a release-note line, not just the PR body.
2. Pre-merge check worth one live run: pods ship runpodctl with pod-scoped API keys. If a pod-scoped key can hit REST /pods but not graphql myPods, every in-pod pod list becomes permanent unknown/runtime_unavailable plus a possible repeated 5s stall per invocation. It degrades rather than breaks, but one test with a pod-scoped key before merge would settle it.
3. Minor: uptimeSeconds is now absent where it used to be a constant (garbage) 0 — consumers reading it as always-present get null; --status still filters the RUNNING/EXITED vocabulary while the output now also speaks running/initializing/stopped, so --status initializing silently matches nothing (a help-text sentence would cover it); the e2e transition test hard-requires observing initializing, which a fast cached-image boot can flake.