Skip to content

Backport release/v6.7: seidb: add changelog mode and --inspect-plan to speed up EVM digest - #4439

Merged
blindchaser merged 1 commit into
release/v6.7from
backport-4419-to-release/v6.7
Oct 3, 2026
Merged

blindchaser merged 1 commit into
release/v6.7from
backport-4419-to-release/v6.7

Conversation

@seidroid

@seidroid seidroid Bot commented Oct 3, 2026

Copy link
Copy Markdown

Backport of #4419 to release/v6.7.

@seidroid

seidroid Bot commented Oct 3, 2026

Copy link
Copy Markdown
Author

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-4419-to-release/v6.7
git worktree add --checkout .worktree/backport-4419-to-release/v6.7 backport-4419-to-release/v6.7
cd .worktree/backport-4419-to-release/v6.7
git reset --hard HEAD^
git cherry-pick -x 53168c2d1ebc10ba3dd0b237b8be24f25b4657aa
git push --force-with-lease

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedOct 3, 2026, 2:16 AM

@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.86441% with 107 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.89%. Comparing base (b9b9f14) to head (ce192ba).

Files with missing lines Patch % Lines
.../seidb/operations/evmlogical/evm_logical_digest.go 84.47% 25 Missing and 18 partials ⚠️
sei-db/state_db/sc/memiavl/changelog_range.go 73.19% 13 Missing and 13 partials ⚠️
.../operations/evmlogical/evm_logical_inspect_plan.go 82.39% 15 Missing and 10 partials ⚠️
...rations/evmlogical/evm_logical_digest_changelog.go 84.00% 5 Missing and 3 partials ⚠️
sei-db/tools/cmd/seidb/main.go 0.00% 2 Missing ⚠️
sei-db/tools/cmd/seidb/operations/dump_flatkv.go 83.33% 1 Missing ⚠️
...cmd/seidb/operations/evmlogical/kvrepair_export.go 80.00% 1 Missing ⚠️
...db/tools/cmd/seidb/operations/flatkv_state_size.go 88.88% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@               Coverage Diff                @@
##           release/v6.7    #4439      +/-   ##
================================================
- Coverage         61.49%   60.89%   -0.61%     
================================================
  Files              2168     2093      -75     
  Lines            189616   182624    -6992     
================================================
- Hits             116607   111208    -5399     
+ Misses            62232    61183    -1049     
+ Partials          10777    10233     -544     
Flag Coverage Δ
sei-chain-pr 53.04% <83.56%> (?)
sei-db 69.80% <ø> (ø)
sei-db-state-db ?
sei-db-state-db-pr 77.29% <73.19%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/tools/cmd/seidb/operations/flatkv_open.go 64.51% <100.00%> (+0.38%) ⬆️
sei-db/tools/cmd/seidb/operations/dump_flatkv.go 50.98% <83.33%> (-0.24%) ⬇️
...cmd/seidb/operations/evmlogical/kvrepair_export.go 81.01% <80.00%> (ø)
...db/tools/cmd/seidb/operations/flatkv_state_size.go 63.86% <88.88%> (ø)
sei-db/tools/cmd/seidb/main.go 0.00% <0.00%> (ø)
...rations/evmlogical/evm_logical_digest_changelog.go 84.00% <84.00%> (ø)
.../operations/evmlogical/evm_logical_inspect_plan.go 82.39% <82.39%> (ø)
sei-db/state_db/sc/memiavl/changelog_range.go 73.19% <73.19%> (ø)
.../seidb/operations/evmlogical/evm_logical_digest.go 73.43% <84.47%> (ø)

... and 83 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…4419)

The kvrepair drill-down runs `seidb evm-logical-digest` many times at
one height on a reserve and on a production node: one digest, then one
`--inspect-bucket --list` report for each bucket and key prefix. The
comparison height is almost never a memIAVL snapshot height, so each run
uses `--memiavl-open-mode replay`. That mode opens the full DB, catches
up every module through the changelog, and walks the EVM tree with
random reads, and it does this again for each report. On a mainnet node
the gap between snapshots is many thousands of blocks, so the drill-down
is slow.

This PR adds two flags and does not change any report value.
`--memiavl-open-mode changelog` scans the `kvs` file of the newest
snapshot at or below `--height` and merges in a sorted overlay of the
EVM writes from the changelog versions above it
(`memiavl.ReplayTreeChangelog`, `scanMemiavlChangelogEVMLeaves`). It
reports the same rows and the same mode labels as replay
(`semantic-replay`, `translator-replay`), so `kvrepair-export` accepts
its reports. It refuses a changelog that does not reach the height, that
starts above the snapshot, or that upgrades the `evm` tree in that
range. The stream opens the snapshot `kvs` file before the scan, so a
prune by the running node does not remove the data under it. When the
node prunes the selected snapshot before the stream opens, the error
wraps `memiavl.ErrSnapshotPruned`, and changelog mode selects the
snapshot again one time. `--inspect-plan` reads a JSON list of inspect
reports and takes all of them from one scan (`inspectFanout`); each
report is byte-equal to the report of a single run with the same flags.
It refuses two reports that write one file, by device and inode, so a
relative path, a symlink, or a hard link does not get past the check. To
make this possible, the memiavl and composite paths now go through one
leaf stream (`memiavlLeafStream`), account inspect lists come out in key
order (before, `--list-limit` kept a subset in map order), and
`entryHash` no longer allocates for account and storage rows.

The EVM logical tools (`evm-logical-digest`, `--inspect-plan`, and
`kvrepair-export`) move into a new package,
`sei-db/tools/cmd/seidb/operations/evmlogical`. The command names and
flags do not change. The FlatKV bucket names and the physical-key
classifier stay in `operations` and are exported, so both packages use
the same copy. The `evmlogical` tests have their own copy of some small
test-data helpers (`addrN`, `noncePair`, `newTestMemiavlStore`, and
others), because there is no shared test-helper package for
`operations`.

The first commit adds golden tests that pin the digest and inspect JSON
for every backend, normalization, and open mode on the current code. No
later commit changes an old golden case. The only output text that
changes is descriptive: the `normalization` field of semantic and
translator reports in replay mode now has the same text as in snapshot
mode ("raw memiavl EVM keys" in place of "replayed memiavl EVM keys"),
and changelog reports have their own `source` text. Review
`changelog_range.go` most closely, mainly the version and upgrade
checks. The tool is read-only and has no consensus or state impact.

Validation: ramtest (also with `-race`) for `sei-db/tools/...` and
`sei-db/state_db/sc/memiavl/...`, with tests that compare changelog mode
to replay mode at many heights, including the kvrepair export end-to-end
test in both modes. On a harbor chain (4 composite validators and 1
memIAVL reserve, snapshot at 20000), every changelog report was equal to
the replay report at heights 15000 to 28000: 74 pairs of digest, bucket
list, and shard reports over two runs. The reserve and composite digests
at 28000 were equal, and `kvrepair-export` accepted the changelog
reports. The EVM overlay was about 1 key for each block on that chain
(8,260 keys, 83 KB for 8,000 blocks). On two pacific-1 shadow nodes at
height 235215000, a memIAVL-only node 85,000 blocks above its snapshot
and a node with a complete migration, the changelog and replay reports
were equal on each node, the two nodes had the same final digest over
707,583,826 rows, and the `--inspect-plan` reports were byte-equal to
single runs. On the memIAVL node, the changelog digest took 405 s with a
24 GB peak RSS, and the replay digest took 2,286 s with a 142 GB peak
RSS. A narrow storage list took 120 s, and 10,052 s with replay.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 53168c2)

Backport to release/v6.7. Conflicts resolved against v6.7, which has no
EVM balance key and no gigatypes.LiveStateStore:
- the FlatKV store type stays flatkv.Store, and the backend constants stay;
- the memiavl EVMKeyBalance decoding and its account bucket entry are dropped,
  and kvrepair-export keeps refusing a balance difference;
- the golden fixture writes nonces where main writes balance rows, and the
  golden reports are recorded on v6.7;
- the translator inspect sorts each translated batch, since v6.7's
  ImportTranslator returns a batch in map order.
@blindchaser
blindchaser force-pushed the backport-4419-to-release/v6.7 branch from 8a5e660 to ce192ba Compare October 3, 2026 02:12
@blindchaser
blindchaser marked this pull request as ready for review October 3, 2026 02:14
@cursor

cursor Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
New memiavl changelog replay semantics and operational tooling paths affect how EVM state is read for comparisons and repairs; mistakes could mis-report digests or repair diffs, though behavior is heavily test-gated.

Overview
Adds ReplayTreeChangelog in memiavl so tools can stream a single tree’s WAL changesets from the same snapshot OpenDB would use, without loading trees or repairing the changelog. It validates version spans, rejects tree upgrades in range, and surfaces ErrSnapshotPruned when the base snapshot disappears mid-read.

seidb evm-logical-digest gains --memiavl-open-mode changelog: sequential snapshot kvs scan merged with EVM changelog writes above it (same logical rows as replay, without walking other modules or the mmap tree). Changelog mode uses the new replay API plus overlay merge; snapshot reads keep kvs open so pruning during a scan can be retried once. EVM logical commands move into the evmlogical package; FlatKV bucket classification is exported from operations for shared use.

Inspect workflows add --inspect-plan: one scan writes many JSON inspect reports from a JSON item list (replacing repeated single-flag runs). kvrepair-export docs now recommend changelog mode for reserve inspects. Golden tests pin digest/inspect JSON across backends and modes; changelog vs replay parity is asserted in tests.

Reviewed by Cursor Bugbot for commit ce192ba. Bugbot is set up for automated code reviews on this repo. Configure here.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Faithful backport of #4419 (offline seidb tooling: memiavl changelog-range replay, changelog open mode, --inspect-plan, evmlogical package split). It differs from upstream in only two places, both correct for release/v6.7: the EVMKeyBalance prefix filter and its test row are dropped because the release branch has no separate balance key, and inspect output is sorted by key so the translator path is deterministic. The dependencies it relies on (wal.Config.NoRepairOnOpen, computeWALIndexDelta, seekSnapshot, utils.NextVersion) exist on the base branch, and I found no issues.

Findings: 0 blocking | 0 non-blocking | 0 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • None at the file/PR level.

@blindchaser
blindchaser requested a review from yzang2019 October 3, 2026 02:46
@blindchaser
blindchaser enabled auto-merge (squash) October 3, 2026 02:47
@blindchaser
blindchaser merged commit a37a6bd into release/v6.7 Oct 3, 2026
80 of 81 checks passed
@blindchaser
blindchaser deleted the backport-4419-to-release/v6.7 branch October 3, 2026 02:52
masih pushed a commit that referenced this pull request Oct 5, 2026
## Summary
- Bump `version.json` from `v6.7.0-rc4` to `v6.7.0` to cut the final
`v6.7` release for mainnet. There is no rc5: this goes straight from rc4
to the release. (v6.6.0 was tagged on the rc5 bump, #3783, with
`version.json` still reading `v6.6.0-rc5`; this keeps `version.json` in
step with the tag.)

Contents since rc4: #4442 and #4439, plus the changelog update (#4443).
Merge this after #4443's backport has landed on `release/v6.7`, so the
`v6.7.0` tag includes the updated changelog.

- Both are labeled `non-app-hash-breaking` (a FlatKV migration metrics
fix and `seidb` digest tooling), so moving from rc4 to `v6.7.0` should
not need a coordinated validator switch.
- Push `v6.7.0` by hand on this PR's merge commit once it has merged;
the tagging ruleset stops `uci-release-publish` from creating it.

## Test plan
- [x] `git diff --check`

Made with [Cursor](https://cursor.com)

Co-authored-by: Cursor <cursoragent@cursor.com>
revofusion pushed a commit to revofusion/sei-chain that referenced this pull request Oct 5, 2026
Adds the `release/v6.7` entries merged since the rc4 changelog (sei-protocol#4406),
in prep to cut **v6.7.0**, the mainnet release (there is no rc5; the
version bump is sei-protocol#4444):

- [sei-protocol#4442](sei-protocol#4442) —
fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes
- [sei-protocol#4439](sei-protocol#4439) — seidb:
add changelog mode and --inspect-plan to speed up EVM digest
- [sei-protocol#4416](sei-protocol#4416) — removal
of the conflict markers the rc4 changelog backport left on
`release/v6.7`
- [sei-protocol#4415](sei-protocol#4415) — rc4
changelog backport
- [sei-protocol#4407](sei-protocol#4407) — rc4
version bump

Regenerated with `./scripts/generate-changelog.sh release/v6.6
release/v6.7`; only the `## v6.7` PR list changes, so the `backport
release/v6.7` cherry-pick applies cleanly (simulated with `git
merge-tree` against `origin/release/v6.7`). Docs-only; no code change.

Made with [Cursor](https://cursor.com)

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants