Repository navigation
fix(telemetry): flush a final heartbeat on graceful shutdown - #1037
Merged
Merged
Conversation
Telemetry counters are in-memory and reset only after an accepted (2xx) send, so everything recorded after the 5-minute first heartbeat was lost unless the process survived to the 24h tick — which most desktop sessions do not. Deferred follow-up from #1029. Stop() now performs ONE final heartbeat after joining the loop: - It reuses sendHeartbeat's send+reset path verbatim, so the counter contract is unchanged — a failed/timed-out flush drops nothing, an accepted flush cannot double-count with the next start. - It runs AFTER the loop is joined rather than from a second goroutine, so there is provably no concurrent sender and the flush cannot interleave with a tick. - It uses a fresh context hard-bounded to 4s (Runtime.Close cancels the app context before Stop, and a dead endpoint must never hang a quit). - It is skipped when telemetry is disabled by env/config, the build is non-semver, the user opted out, or nothing was recorded since the last accepted send (new CounterRegistry.HasPendingCounters). - Anonymous-id rotation is suppressed on this path: rotation rewrites the whole config file, and shutdown is the wrong moment for that. Also closes a pre-existing data race the flush widens: s.endpoint is rewritten under s.mu by NotifyConfigChanged but was read unlocked by the heartbeat, feedback, and opt-out-beacon paths. All four now read it through liveEndpoint(), matching liveConfig().
Deploying mcpproxy-docs with
|
| Latest commit: |
a854ee7
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://68e624f0.mcpproxy-docs.pages.dev |
| Branch Preview URL: | https://fix-telemetry-shutdown-flush.mcpproxy-docs.pages.dev |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Cross-model review round 1 (opencode / gpt-5.6-sol) found three real defects in the graceful-shutdown flush. 1. A failed flush destroyed the installer attribution. resolveLaunchSource clears installer_heartbeat_pending at BUILD time, before the send is known to have landed — a deliberate trade for the long-running loop, but it inverts on the flush: the process is exiting, so an offline or timed-out flush spent a one-shot that previously survived into the next run. The flush now leaves the flag armed and reports the runtime-detected source; the next start still claims "installer" exactly once. Generalized the rotate flag into consumeOneShots, which now gates both the 365-day anonymous-id rotation and the installer flag (send/buildHeartbeat* renamed accordingly). 2. TestFlushOnStopNoDoubleSendWithPeriodicTick was flaky — reproduced at -count=300 (got 4, want 2). Its premise was wrong: a tick aborted after the endpoint accepted it leaves counters pending, so the flush re-sends them. Delivery is at-least-once (undecidable client-side), which is the deliberate trade against dropping the window entirely. Split into two deterministic tests: the flush skips after an ACCEPTED tick, and joins + re-sends an unconfirmed one. Documented the semantics in code and docs. 3. TestFlushOnStopSkippedWhenOptedOut raced the opt-out beacon, which POSTs to the same /heartbeat ingest path (its comment claimed /opt-out). The recorder now classifies beacons by their `event` field and the test waits for the beacon before asserting no usage heartbeat followed. Rejected as pre-existing/undecidable, with rebuttals: heartbeat-URL validateTelemetryURL pinning (pre-existing sink, local config value), buildHeartbeat not bounded by flushTimeout (the joined tick already had the same exposure), and the opt-out latch race (the flush's telemetryEnabledLive plus the pre-transmit optedOut re-check already close the described window).
Contributor
📦 Build ArtifactsWorkflow Run: View Run Available Artifacts
How to DownloadOption 1: GitHub Web UI (easiest)
Option 2: GitHub CLI gh run download 32765967430 --repo smart-mcp-proxy/mcpproxy-go
|
Cross-model review round 2 (opencode / gpt-5.6-sol). One genuine finding: the second half of this PR's own s.endpoint drive-by was left undone. EmitOptOutBeacon and SendOptOutBeacon both gate on s.config.GetAnonymousID() read WITHOUT s.mu, two lines above the endpoint read that now takes it. NotifyConfigChanged swaps s.config under s.mu, and GetAnonymousID dereferences cfg.Telemetry — a pointer ensureAnonymousIDOnce / advanceUpgradeFunnelOnce also install under s.mu — so snapshotting the pointer would not have been enough either; the whole read has to be inside the critical section, exactly as telemetryEnabledLive documents. NotifyConfigChanged itself launches the beacon goroutine, so the writer and the reader are one call apart. Added liveAnonymousID() alongside liveConfig()/liveEndpoint() and routed both gates through it. TestOptOutBeaconReadsAnonymousIDUnderLock reproduces the race first (the detector flags both sites against optout.go:215) and passes after. No unlocked s.config read remains outside a locked region. Also re-raised and rejected again: the opt-out TOCTOU between the pre-transmit optedOut check and client.Do. It is pre-existing on the periodic sender and unchanged in character here, and the proposed fix — an emission mutex held across the HTTP send — would make NotifyConfigChanged block the config-save path on up to 10s of network I/O, breaking the "never blocks the caller" invariant that goroutine exists to uphold.
This was referenced Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Telemetry counters live in memory and are reset only after an accepted (2xx) send. There was no send on shutdown, so everything recorded after the 5-minute first heartbeat was discarded unless the process survived to the 24h tick — which most desktop sessions don't. Deferred follow-up from #1029.
Service.Stop()now performs one final heartbeat, after it has joined the heartbeat loop.Key decisions
sendHeartbeatbody, so the counter contract is untouched: a failed or timed-out flush resets nothing (counts survive to the next run), and an accepted flush can't double-count with a later start.Runtime.Closebefore the shutdown marker resolves and the BBolt handle closes, which is wherebuildHeartbeat's funnel writes are legal.Runtime.Closecancels the app context beforeStop, so a derived context would abort instantly; and a dead endpoint must never delay a quit.CounterRegistry.HasPendingCounters, inspecting exactly the setResetzeroes).consumeOneShots=false). Two side effects are consumed when the payload is built rather than when it is accepted: the 365-day anonymous-id rotation (which rewrites the whole config file) and theinstaller_heartbeat_pendingflag. Spending either for a send that may never land would destroy state a normal run still has, so the flush skips both and the next start performs them. Every other build-time touch (activationStore.Load, the diagnostics/preflight snapshots) is adb.View, andfunnelStore.RecordActivityis idempotent day-marking.docs/features/telemetry.md— receivers dedup per(anonymous_id, timestamp).liveConfig()snapshot,persistConfigliveness guard, atomic rotation unders.mu); no races reintroduced.Drive-by: two unlocked reads of state
NotifyConfigChangedwritesNotifyConfigChangedrewrites boths.endpointands.configunders.mu, and it is one call away from the beacon goroutine it launches.s.endpointwas read unlocked by the heartbeat, feedback, and opt-out-beacon paths — a race the flush widens, since it now sends from whichever goroutine calledStop. All four reads go through a newliveEndpoint().s.config.GetAnonymousID()was read unlocked by both opt-out-beacon eligibility gates, two lines above that endpoint read. Snapshotting the pointer would not have sufficed:GetAnonymousIDdereferencescfg.Telemetry, whichensureAnonymousIDOnce/advanceUpgradeFunnelOncealso install unders.mu, so the whole read has to be inside the critical section — exactly astelemetryEnabledLivedocuments. NewliveAnonymousID().No unlocked
s.configread remains outside a locked region.Tests
internal/telemetry/shutdown_flush_test.go: sent-once + counters reset, post-first-heartbeat activity captured (the regression), flush timeout leaves shutdown unblocked and counters retained, opted-out and disabled installs send nothing, the flush skips after an already-accepted tick, it joins + re-sends an unconfirmed one, and the anonymous-id read is locked (fails under-racewithoutliveAnonymousID, flagging both call sites againstoptout.go:215).activation_test.goadditionally pins that a non-consuming resolve leavesinstaller_heartbeat_pendingarmed.go test -racepasses for./internal/telemetry/...,./internal/runtime/...,./internal/httpapi/...,./internal/server/(CI skip regex),./internal/config|storage|index|upstream/...; the flush tests survive-count=200 -race.golangci-lintv2 with.github/.golangci.ymlreports 0 issues. (TestScanForPII_V7FieldViolationsfails at-count=2on this machine onorigin/maintoo — the local username is literallyuser, a blocked value that leaks into the second pass viaPopulateBlockedValues; verified pre-existing, not a regression.)Live verification against a real binary (
-ldflagssemver, isolated--config/--data-dir, telemetry endpoint pointed at a local sink, port 18088):SIGTERM→ nothing before shutdown (the loop is parked in the 5-minute initial delay), then exactly onePOST /heartbeatcarryingsurface_requests.unknown: 11— the session that used to be dropped.launch_sourcecame back as the runtime-detectedunknown, not a consumed one-shot.DO_NOT_TRACK=1→ sink stays empty; the log showsTelemetry disabled by environment variable, and the flush is never armed.CI note
The first
Unit Tests (ubuntu-latest, 1.25)run failed onTestFastPathDoesNotOvertakeAQueue(internal/upstream/limiter/limiter_test.go:635, "FIFO violated") — a package this PR does not touch at all. Verified pre-existing: it reproduces on a cleanorigin/mainworktree atgo test ./internal/upstream/limiter/ -run TestFastPathDoesNotOvertakeAQueue -count=400. Rerun requested; worth a real fix on main, since the assertion demands strict FIFO from a fast path that only usually loses the race.Cross-model review
Reviewed with the opencode CLI (gpt-5.6-sol). Round 1 and round 2 both returned CHANGES REQUIRED; every finding below was verified against the code before being acted on.
Round 1 — three genuine defects, all fixed:
resolveLaunchSourceclearsinstaller_heartbeat_pendingat build time; on the shutdown path an offline/timed-out flush destroyed an attribution that previously survived into the next run.TestFlushOnStopNoDoubleSendWithPeriodicTickwas flaky — reproduced locally at-count=300(got 4, want 2). Its premise ("exactly once whatever the interleaving") was simply not what the design guarantees. Split into two deterministic tests plus the at-least-once documentation.TestFlushOnStopSkippedWhenOptedOutraced the opt-out beacon, which POSTs to the same/heartbeatingest path (the test comment claimed/opt-out). Verified empirically. The recorder now classifies beacons by theireventfield and the test waits for the beacon before asserting.Round 2 — one genuine defect, fixed: the unlocked
GetAnonymousIDreads described under Drive-by above, reproduced under-racebefore the fix.Round 3 — CLEAN. "No new correctness or security defects found. Applied fixes are complete and consistent with shutdown ordering."
Rejected, with rebuttals:
validateTelemetryURL. Pre-existing sink; the endpoint is a local config value documented as a testing aid. Worth its own PR, not this one.buildHeartbeatis not bounded byflushTimeout. The pre-existing joined tick has the identical exposure, andRuntime.Closehas no hard deadline (upstream shutdown alone budgets 45s+15s ahead of it).optedOutcheck andclient.Do(raised twice). Pre-existing on the periodic sender and unchanged in character here; the flush'stelemetryEnabledLive()closes the window the round-1 report described. The proposed fix — an emission mutex held across the HTTP send — would makeNotifyConfigChangedblock the config-save path on up to 10s of network I/O, breaking the "never blocks the caller" invariant that goroutine exists to uphold.