Skip to content

Backport release/v6.7: fix(seidb): report only the current migration boundary on the snapshot gauge - #4348

Merged
alexander-sei merged 2 commits into
release/v6.7from
backport-4327-to-release/v6.7
Sep 25, 2026
Merged

alexander-sei merged 2 commits into
release/v6.7from
backport-4327-to-release/v6.7

Conversation

@seidroid

@seidroid seidroid Bot commented Sep 25, 2026

Copy link
Copy Markdown

Backport of #4327 to release/v6.7.

…t gauge (#4327)

`seidb_migration_boundary_snapshot` grows without bound during a
migration. Every 10 seconds `MigrationMetrics` recorded the current
boundary as a new `boundary_hex` label on a synchronous OTel gauge, and
the OTel Go SDK keeps every attribute set a synchronous gauge has
recorded and exports all of them on each scrape. On arctic-1, each
migrating node exports about 15.7k series for this metric, growing by
360 per hour, which is about 615k active series in prod Prometheus.
Because every series has the value 1, no query can tell which one is the
current boundary, so the metric cannot serve its purpose either.

The boundary snapshot is now an `Int64ObservableGauge`.
`registerBoundarySnapshot` registers a callback,
`observeBoundarySnapshot`, that reports only the current boundary at
each collection, or `complete` once the version reaches `targetVersion`.
The callback is unregistered when the router context is cancelled, so a
rebuilt or closed router, including a read-only handle, stops reporting.
This removes the 10-second snapshot loop and the interval parameter of
`NewMigrationMetrics`. The metric name, value, and label stay the same.

There is no consensus, state, or wire-format impact; the change is
limited to metrics. New tests in `migration_metrics_test.go` use a
manual reader to check that only the current boundary is exported, that
the label is `complete` at the target version, and that nothing is
exported after `Close`. The `migration`, `composite`, and `rootmulti`
package tests pass.

Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit c6ed07f)
@cursor

cursor Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Observability-only change to migration metrics emission; migration routing and persistence behavior are unchanged.

Overview
Fixes migration boundary Prometheus cardinality by replacing the periodic synchronous seidb_migration_boundary_snapshot gauge (which retained every historical boundary_hex label) with an OpenTelemetry Int64ObservableGauge that reports only the current boundary on each scrape.

NewMigrationMetrics no longer takes a snapshot interval; the 10s ticker loop and recordBoundarySnapshot writes are gone. A RegisterCallback observer reads in-memory boundary/version state, emits boundary_hex as the live boundary or complete at targetVersion, and Close cancels context so a small goroutine unregisters the callback. Router builders and comments are updated accordingly.

New tests assert a single label per collect, the complete sentinel at target version, and no export after Close.

Reviewed by Cursor Bugbot for commit 23b340c. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedSep 25, 2026, 7:58 PM

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a clean backport of #4327. It swaps the synchronous boundary-snapshot gauge and its ticker goroutine for an observable gauge with a registered callback, so each collection reports only the current boundary_hex label (or complete). The callback is unregistered when ctx is cancelled or Close is called. All three NewMigrationMetrics call sites are updated, and the new ManualReader tests cover the current-boundary, complete, and post-Close behavior. I found nothing blocking; I couldn't run the tests here because running them needed approval.

Findings: 0 blocking | 0 non-blocking | 0 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • None at the file/PR level.

@codecov

codecov Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.95652% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.54%. Comparing base (d756203) to head (23b340c).

Files with missing lines Patch % Lines
sei-db/state_db/sc/migration/migration_metrics.go 85.00% 1 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##           release/v6.7    #4348      +/-   ##
================================================
- Coverage         61.37%   60.54%   -0.84%     
================================================
  Files              2163     2081      -82     
  Lines            189085   179231    -9854     
================================================
- Hits             116059   108513    -7546     
+ Misses            62301    60786    -1515     
+ Partials          10725     9932     -793     
Flag Coverage Δ
sei-db 69.80% <ø> (-0.22%) ⬇️
sei-db-state-db ?
sei-db-state-db-pr 79.12% <86.95%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/state_db/sc/composite/store.go 70.48% <ø> (ø)
sei-db/state_db/sc/migration/migration_manager.go 95.53% <ø> (ø)
sei-db/state_db/sc/migration/router_builder.go 57.71% <100.00%> (ø)
sei-db/state_db/sc/migration/migration_metrics.go 90.53% <85.00%> (+0.75%) ⬆️

... and 83 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alexander-sei
alexander-sei enabled auto-merge (squash) September 25, 2026 18:37
@alexander-sei
alexander-sei enabled auto-merge (squash) September 25, 2026 19:57
@alexander-sei
alexander-sei merged commit 0d56aae into release/v6.7 Sep 25, 2026
72 checks passed
@alexander-sei
alexander-sei deleted the backport-4327-to-release/v6.7 branch September 25, 2026 20:19
alexander-sei added a commit that referenced this pull request Sep 25, 2026
Regenerated with ./scripts/generate-changelog.sh release/v6.6 release/v6.7.

Co-authored-by: Cursor <cursoragent@cursor.com>
masih pushed a commit that referenced this pull request Oct 1, 2026
Adds the `release/v6.7` entries merged since the rc2 changelog (#4293),
in prep to cut **v6.7.0-rc4**. The rc3 update (#4335) was closed without
merging, so its entries are included here:

- [#4411](#4411) — Log a
pinned node's skipped migration kick-off once per batch size
- [#4410](#4410) — Apply
compiled KV repair files at a fixed height and generate them from digest
inspect lists
- [#4409](#4409) — feat:
add migration pause handler
- [#4397](#4397) —
fix(evmrpc): release eth_getLogs DB-read slots when a block read panics
- [#4378](#4378) — Raise
goreleaser timeout to 2h
- [#4377](#4377) — Fix
FlatKV state sync bad-hash scenario
- [#4371](#4371) —
fix(seidb): keep writes in the old DB until the migration boundary first
moves
- [#4348](#4348) —
fix(seidb): report only the current migration boundary on the snapshot
gauge
- [#4347](#4347) —
fix(flatkv): keep 10 old checkpoints instead of mirroring memIAVL's
count
- [#4339](#4339) — rc3
version bump
- [#4334](#4334) — Fail
dynamic-gas precompile out-of-gas as an EVM out-of-gas call
- [#4315](#4315) — Pin the
Go builder image per architecture in build-static.sh
- [#4313](#4313) —
fix(memiavl): hold a snapshot reference for an iterator's lifetime
- [#4295](#4295) — rc2
version bump
- [#4294](#4294) — rc2
changelog backport

Also returns `## v6.7` to the format used through v6.6: the version
heading, `sei-chain`, and the generated PR list. The hand-written `###
Improvements` and `### Upgrade guide` sections are removed; every PR
they described is already a line in the generated list.

Regenerated with `./scripts/generate-changelog.sh release/v6.6
release/v6.7`; only the `## v6.7` section changes. Docs-only; no code
change.

**Backport note:** the #4347 backport added its own line to the top of
the `## v6.7` list on `release/v6.7`, and `main` doesn't have it, so the
`backport release/v6.7` cherry-pick of this PR conflicts at that one
spot (simulated with `git merge-tree`). Resolve it by taking this PR's
side: its fifteen lines already include #4347.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants