Repository navigation
Backport release/v6.7: seidb: add changelog mode and --inspect-plan to speed up EVM digest - #4439
Conversation
|
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-4419-to-release/v6.7
git worktree add --checkout .worktree/backport-4419-to-release/v6.7 backport-4419-to-release/v6.7
cd .worktree/backport-4419-to-release/v6.7
git reset --hard HEAD^
git cherry-pick -x 53168c2d1ebc10ba3dd0b237b8be24f25b4657aa
git push --force-with-lease |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release/v6.7 #4439 +/- ##
================================================
- Coverage 61.49% 60.89% -0.61%
================================================
Files 2168 2093 -75
Lines 189616 182624 -6992
================================================
- Hits 116607 111208 -5399
+ Misses 62232 61183 -1049
+ Partials 10777 10233 -544
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
…4419) The kvrepair drill-down runs `seidb evm-logical-digest` many times at one height on a reserve and on a production node: one digest, then one `--inspect-bucket --list` report for each bucket and key prefix. The comparison height is almost never a memIAVL snapshot height, so each run uses `--memiavl-open-mode replay`. That mode opens the full DB, catches up every module through the changelog, and walks the EVM tree with random reads, and it does this again for each report. On a mainnet node the gap between snapshots is many thousands of blocks, so the drill-down is slow. This PR adds two flags and does not change any report value. `--memiavl-open-mode changelog` scans the `kvs` file of the newest snapshot at or below `--height` and merges in a sorted overlay of the EVM writes from the changelog versions above it (`memiavl.ReplayTreeChangelog`, `scanMemiavlChangelogEVMLeaves`). It reports the same rows and the same mode labels as replay (`semantic-replay`, `translator-replay`), so `kvrepair-export` accepts its reports. It refuses a changelog that does not reach the height, that starts above the snapshot, or that upgrades the `evm` tree in that range. The stream opens the snapshot `kvs` file before the scan, so a prune by the running node does not remove the data under it. When the node prunes the selected snapshot before the stream opens, the error wraps `memiavl.ErrSnapshotPruned`, and changelog mode selects the snapshot again one time. `--inspect-plan` reads a JSON list of inspect reports and takes all of them from one scan (`inspectFanout`); each report is byte-equal to the report of a single run with the same flags. It refuses two reports that write one file, by device and inode, so a relative path, a symlink, or a hard link does not get past the check. To make this possible, the memiavl and composite paths now go through one leaf stream (`memiavlLeafStream`), account inspect lists come out in key order (before, `--list-limit` kept a subset in map order), and `entryHash` no longer allocates for account and storage rows. The EVM logical tools (`evm-logical-digest`, `--inspect-plan`, and `kvrepair-export`) move into a new package, `sei-db/tools/cmd/seidb/operations/evmlogical`. The command names and flags do not change. The FlatKV bucket names and the physical-key classifier stay in `operations` and are exported, so both packages use the same copy. The `evmlogical` tests have their own copy of some small test-data helpers (`addrN`, `noncePair`, `newTestMemiavlStore`, and others), because there is no shared test-helper package for `operations`. The first commit adds golden tests that pin the digest and inspect JSON for every backend, normalization, and open mode on the current code. No later commit changes an old golden case. The only output text that changes is descriptive: the `normalization` field of semantic and translator reports in replay mode now has the same text as in snapshot mode ("raw memiavl EVM keys" in place of "replayed memiavl EVM keys"), and changelog reports have their own `source` text. Review `changelog_range.go` most closely, mainly the version and upgrade checks. The tool is read-only and has no consensus or state impact. Validation: ramtest (also with `-race`) for `sei-db/tools/...` and `sei-db/state_db/sc/memiavl/...`, with tests that compare changelog mode to replay mode at many heights, including the kvrepair export end-to-end test in both modes. On a harbor chain (4 composite validators and 1 memIAVL reserve, snapshot at 20000), every changelog report was equal to the replay report at heights 15000 to 28000: 74 pairs of digest, bucket list, and shard reports over two runs. The reserve and composite digests at 28000 were equal, and `kvrepair-export` accepted the changelog reports. The EVM overlay was about 1 key for each block on that chain (8,260 keys, 83 KB for 8,000 blocks). On two pacific-1 shadow nodes at height 235215000, a memIAVL-only node 85,000 blocks above its snapshot and a node with a complete migration, the changelog and replay reports were equal on each node, the two nodes had the same final digest over 707,583,826 rows, and the `--inspect-plan` reports were byte-equal to single runs. On the memIAVL node, the changelog digest took 405 s with a 24 GB peak RSS, and the replay digest took 2,286 s with a 142 GB peak RSS. A narrow storage list took 120 s, and 10,052 s with replay. --------- Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit 53168c2) Backport to release/v6.7. Conflicts resolved against v6.7, which has no EVM balance key and no gigatypes.LiveStateStore: - the FlatKV store type stays flatkv.Store, and the backend constants stay; - the memiavl EVMKeyBalance decoding and its account bucket entry are dropped, and kvrepair-export keeps refusing a balance difference; - the golden fixture writes nonces where main writes balance rows, and the golden reports are recorded on v6.7; - the translator inspect sorts each translated batch, since v6.7's ImportTranslator returns a batch in map order.
8a5e660 to
ce192ba
Compare
PR SummaryMedium Risk Overview
Inspect workflows add Reviewed by Cursor Bugbot for commit ce192ba. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Faithful backport of #4419 (offline seidb tooling: memiavl changelog-range replay, changelog open mode, --inspect-plan, evmlogical package split). It differs from upstream in only two places, both correct for release/v6.7: the EVMKeyBalance prefix filter and its test row are dropped because the release branch has no separate balance key, and inspect output is sorted by key so the translator path is deterministic. The dependencies it relies on (wal.Config.NoRepairOnOpen, computeWALIndexDelta, seekSnapshot, utils.NextVersion) exist on the base branch, and I found no issues.
Findings: 0 blocking | 0 non-blocking | 0 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- None at the file/PR level.
## Summary - Bump `version.json` from `v6.7.0-rc4` to `v6.7.0` to cut the final `v6.7` release for mainnet. There is no rc5: this goes straight from rc4 to the release. (v6.6.0 was tagged on the rc5 bump, #3783, with `version.json` still reading `v6.6.0-rc5`; this keeps `version.json` in step with the tag.) Contents since rc4: #4442 and #4439, plus the changelog update (#4443). Merge this after #4443's backport has landed on `release/v6.7`, so the `v6.7.0` tag includes the updated changelog. - Both are labeled `non-app-hash-breaking` (a FlatKV migration metrics fix and `seidb` digest tooling), so moving from rc4 to `v6.7.0` should not need a coordinated validator switch. - Push `v6.7.0` by hand on this PR's merge commit once it has merged; the tagging ruleset stops `uci-release-publish` from creating it. ## Test plan - [x] `git diff --check` Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com>
Adds the `release/v6.7` entries merged since the rc4 changelog (sei-protocol#4406), in prep to cut **v6.7.0**, the mainnet release (there is no rc5; the version bump is sei-protocol#4444): - [sei-protocol#4442](sei-protocol#4442) — fix(seidb): fix stale FlatKV migration gauges on snapshotting nodes - [sei-protocol#4439](sei-protocol#4439) — seidb: add changelog mode and --inspect-plan to speed up EVM digest - [sei-protocol#4416](sei-protocol#4416) — removal of the conflict markers the rc4 changelog backport left on `release/v6.7` - [sei-protocol#4415](sei-protocol#4415) — rc4 changelog backport - [sei-protocol#4407](sei-protocol#4407) — rc4 version bump Regenerated with `./scripts/generate-changelog.sh release/v6.6 release/v6.7`; only the `## v6.7` PR list changes, so the `backport release/v6.7` cherry-pick applies cleanly (simulated with `git merge-tree` against `origin/release/v6.7`). Docs-only; no code change. Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com>
Backport of #4419 to
release/v6.7.