fix(flatkv): keep 10 old checkpoints instead of mirroring memIAVL's count

[blindchaser]

Summary

A composite read needs a version that both backends still hold. At mainnet state size, memIAVL publishes a snapshot only about every 50,000 blocks. One retained FlatKV checkpoint reaches back only 10,000 to 20,000 blocks, so FlatKV prunes each version before memIAVL publishes it, and no common version exists. On the mainnet shadow pair, memIAVL published 229810000 when FlatKV held only 229840000 and 229850000. This blocks a cross-backend digest and leaves a composite rollback with no shared base snapshot.

Changes

• FlatKV's default SnapshotKeepRecent goes from 1 to 10. At the default 10,000-block interval, this is a guaranteed reach of 100,000 blocks (about 12 hours).
• alignFlatKVSnapshotWithMemIAVL still copies memIAVL's effective snapshot interval to FlatKV, but it no longer overwrites FlatKV's keep-recent with sc-keep-recent. Without this, the new default has no effect on a node (the review finding on fix(flatkv): keep 10 old checkpoints instead of mirroring memIAVL's count #4158).
• New test TestNewCompositeCommitStoreKeepsFlatKVKeepRecent builds a composite store from the default config and checks the effective FlatKV retention. TestAlignFlatKVSnapshotWithMemIAVL now checks that keep-recent is left as configured.
• Three test configs that set only memIAVL keep-recent (and relied on the mirror) now also set FlatKV keep-recent.

Cost

A checkpoint hardlinks its SSTs, so each one costs only what compaction has obsoleted since: about 261 MiB of pinned SSTs plus about 25 MiB of retained state WAL at mainnet state size, roughly 2.8 GiB for ten.

Config

No app.toml change is needed. The node path (parseSCConfigs in app/seidb.go) does not read state-commit.flatkv.snapshot-keep-recent, so a key rendered by the v6.4–v6.6 templates does not pin the old value. Every node gets the new default on restart.

Backport

Replaces #4158, which targeted release/v6.7 directly. Labelled backport release/v6.7. The FlatKV config.go differs between main and release/v6.7 (v6.7 still has the DefaultSnapshotKeepRecent constant and the config golden records), so the automatic backport can conflict there.

Testing

• scripts/ramtest.sh on sei-db/state_db/sc/composite, sei-db/state_db/sc/flatkv/..., sei-db/config, sei-cosmos/storev2/rootmulti, sei-cosmos/server/config, and the app config tests.
• make fmtcheck.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-24T14:11:16.967113Z	d193308	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cursor]

PR Summary

Medium Risk
Changes state-commit snapshot retention on composite nodes (more disk for FlatKV checkpoints) and alters alignment with memIAVL; incorrect retention could still break cross-backend reads or rollback.

Overview
Fixes composite state-commit reads where FlatKV pruned checkpoints before memIAVL exposed a matching snapshot, so memIAVL and FlatKV had no overlapping version for digests and rollback.

FlatKV default SnapshotKeepRecent rises from 1 to 10 (~100k blocks at the 10k interval) so checkpoints outlive memIAVL’s slower snapshot cadence. alignFlatKVSnapshotWithMemIAVL still mirrors memIAVL’s effective snapshot interval but stops overwriting FlatKV keep-recent with sc-keep-recent, so the new default applies on restart. Tests and integration helpers set FlatKV keep-recent explicitly where they used to rely on the mirror.

^{Reviewed by Cursor Bugbot for commit d3c5d2e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 25, 2026, 2:39 PM

[seidroid]

This PR stops alignFlatKVSnapshotWithMemIAVL from copying memIAVL's keep-recent onto FlatKV and raises FlatKV's default from 1 to 10. FlatKV still takes its snapshot interval from memIAVL. The change is small and correct, and the tests that relied on the old copying were updated. The only note is that the fixed count of 10 no longer covers the gap if an operator lowers sc-snapshot-interval.

Findings: 0 blocking | 1 non-blocking | 0 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• [suggestion] FlatKV's reach is now 10 × the snapshot interval, and the interval still comes from sc-snapshot-interval. At mainnet size, memIAVL publishes a snapshot about every 50,000 blocks because writing one is slow, so that gap does not shrink when the interval does. An operator who lowers sc-snapshot-interval (e.g. to 1,000) gets 10,000 blocks of FlatKV reach, and the backends again share no version. Either document this interaction or derive keep-recent from a target block depth (e.g. ceil(100000/interval)) so the stated guarantee holds for any interval.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d193308172

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Pin the 10-checkpoint default in characterization

Add a literal characterization assertion for the new value. The added composite test obtains both the actual and expected values from DefaultStateCommitConfig() and merely checks that FlatKV differs from memIAVL, so changing this default from 10 to another non-1 value would remain green and the configuration change is not recorded as an old-to-new diff as required.

AGENTS.md reference: AGENTS.md:L22-L30

Useful? React with 👍 / 👎.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.51%. Comparing base (6018de1) to head (d3c5d2e).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4322      +/-   ##
==========================================
- Coverage   67.60%   66.51%   -1.10%     
==========================================
  Files        2192     2066     -126     
  Lines      168752   156304   -12448     
==========================================
- Hits       114087   103966   -10121     
+ Misses      54655    52328    -2327     
  Partials       10       10              

Flag	Coverage Δ
sei-chain-pr	84.13% <ø> (?)
sei-db	74.50% <ø> (-0.25%)	⬇️
sei-db-state-db	?
sei-db-state-db-pr	80.75% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-cosmos/server/config/config.go	99.01% <ø> (ø)
sei-db/state_db/sc/composite/store.go	79.06% <100.00%> (-0.04%)	⬇️
sei-db/state_db/sc/flatkv/config/config.go	88.88% <100.00%> (ø)

... and 216 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

Created backport PR for release/v6.7:

• Backport release/v6.7: fix(flatkv): keep 10 old checkpoints instead of mirroring memIAVL's count #4347 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-4322-to-release/v6.7
git worktree add --checkout .worktree/backport-4322-to-release/v6.7 backport-4322-to-release/v6.7
cd .worktree/backport-4322-to-release/v6.7
git reset --hard HEAD^
git cherry-pick -x 8c95ff34abd8634c1bfbabaddc6fbe85823a2c02
git push --force-with-lease