Repository navigation
Backport release/v6.7: fix(seidb): report only the current migration boundary on the snapshot gauge - #4348
Conversation
…t gauge (#4327) `seidb_migration_boundary_snapshot` grows without bound during a migration. Every 10 seconds `MigrationMetrics` recorded the current boundary as a new `boundary_hex` label on a synchronous OTel gauge, and the OTel Go SDK keeps every attribute set a synchronous gauge has recorded and exports all of them on each scrape. On arctic-1, each migrating node exports about 15.7k series for this metric, growing by 360 per hour, which is about 615k active series in prod Prometheus. Because every series has the value 1, no query can tell which one is the current boundary, so the metric cannot serve its purpose either. The boundary snapshot is now an `Int64ObservableGauge`. `registerBoundarySnapshot` registers a callback, `observeBoundarySnapshot`, that reports only the current boundary at each collection, or `complete` once the version reaches `targetVersion`. The callback is unregistered when the router context is cancelled, so a rebuilt or closed router, including a read-only handle, stops reporting. This removes the 10-second snapshot loop and the interval parameter of `NewMigrationMetrics`. The metric name, value, and label stay the same. There is no consensus, state, or wire-format impact; the change is limited to metrics. New tests in `migration_metrics_test.go` use a manual reader to check that only the current boundary is exported, that the label is `complete` at the target version, and that nothing is exported after `Close`. The `migration`, `composite`, and `rootmulti` package tests pass. Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit c6ed07f)
PR SummaryLow Risk Overview
New tests assert a single label per collect, the complete sentinel at target version, and no export after Reviewed by Cursor Bugbot for commit 23b340c. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
There was a problem hiding this comment.
This is a clean backport of #4327. It swaps the synchronous boundary-snapshot gauge and its ticker goroutine for an observable gauge with a registered callback, so each collection reports only the current boundary_hex label (or complete). The callback is unregistered when ctx is cancelled or Close is called. All three NewMigrationMetrics call sites are updated, and the new ManualReader tests cover the current-boundary, complete, and post-Close behavior. I found nothing blocking; I couldn't run the tests here because running them needed approval.
Findings: 0 blocking | 0 non-blocking | 0 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- None at the file/PR level.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/v6.7 #4348 +/- ##
================================================
- Coverage 61.37% 60.54% -0.84%
================================================
Files 2163 2081 -82
Lines 189085 179231 -9854
================================================
- Hits 116059 108513 -7546
+ Misses 62301 60786 -1515
+ Partials 10725 9932 -793
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Regenerated with ./scripts/generate-changelog.sh release/v6.6 release/v6.7. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds the `release/v6.7` entries merged since the rc2 changelog (#4293), in prep to cut **v6.7.0-rc4**. The rc3 update (#4335) was closed without merging, so its entries are included here: - [#4411](#4411) — Log a pinned node's skipped migration kick-off once per batch size - [#4410](#4410) — Apply compiled KV repair files at a fixed height and generate them from digest inspect lists - [#4409](#4409) — feat: add migration pause handler - [#4397](#4397) — fix(evmrpc): release eth_getLogs DB-read slots when a block read panics - [#4378](#4378) — Raise goreleaser timeout to 2h - [#4377](#4377) — Fix FlatKV state sync bad-hash scenario - [#4371](#4371) — fix(seidb): keep writes in the old DB until the migration boundary first moves - [#4348](#4348) — fix(seidb): report only the current migration boundary on the snapshot gauge - [#4347](#4347) — fix(flatkv): keep 10 old checkpoints instead of mirroring memIAVL's count - [#4339](#4339) — rc3 version bump - [#4334](#4334) — Fail dynamic-gas precompile out-of-gas as an EVM out-of-gas call - [#4315](#4315) — Pin the Go builder image per architecture in build-static.sh - [#4313](#4313) — fix(memiavl): hold a snapshot reference for an iterator's lifetime - [#4295](#4295) — rc2 version bump - [#4294](#4294) — rc2 changelog backport Also returns `## v6.7` to the format used through v6.6: the version heading, `sei-chain`, and the generated PR list. The hand-written `### Improvements` and `### Upgrade guide` sections are removed; every PR they described is already a line in the generated list. Regenerated with `./scripts/generate-changelog.sh release/v6.6 release/v6.7`; only the `## v6.7` section changes. Docs-only; no code change. **Backport note:** the #4347 backport added its own line to the top of the `## v6.7` list on `release/v6.7`, and `main` doesn't have it, so the `backport release/v6.7` cherry-pick of this PR conflicts at that one spot (simulated with `git merge-tree`). Resolve it by taking this PR's side: its fifteen lines already include #4347. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
Backport of #4327 to
release/v6.7.