Repository navigation
Conversation
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
PR SummaryHigh Risk Overview The write index is built up front (per-tx, sharded) and conflicts are checked for writes in Docs/README and tests add large mixed-block OCC vs sequential parity, shard/conflict coverage, and an ERC20 loadtest benchmark scenario. Reviewed by Cursor Bugbot for commit 1143167. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## devin/1789943071-stack-3-prepare-pipeline #4273 +/- ##
=============================================================================
+ Coverage 83.02% 84.71% +1.69%
=============================================================================
Files 30 27 -3
Lines 2992 4357 +1365
=============================================================================
+ Hits 2484 3691 +1207
- Misses 507 666 +159
+ Partials 1 0 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
The parallel OCC validation and sharded merge look correct: the pre-built span index is equivalent to the old accept-as-you-go index for the only two sourcePrefix values that occur, stmFrontierAccepts reproduces the serial accept condition, shard ownership is disjoint and totally covering, and shard order preserves canonical address order. Only two non-blocking efficiency observations.
Findings: 0 blocking | 2 non-blocking | 1 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- [suggestion]
occShardOfkeys onaddr[0]only, which is required for order-preserving concatenation but means a single-hot-contract workload puts all of that contract's storage into one shard. Theerc20_single_contractbenchmark this PR adds is exactly that shape, so the sharded merge will parallelize little there — worth calling out in the README alongside the existing shard description so the next reader doesn't expect merge speedup on single-contract blocks. - 1 suggestion(s)/nit(s) flagged inline on specific lines.
| if len(results)-from < occMinParallelValidation { | ||
| return 0, nil | ||
| } | ||
| cumulative, to := cumulativeGasFrom(results, from, state.cumulativeGasUsed) |
There was a problem hiding this comment.
[suggestion] cumulativeGasFrom walks and allocates for the entire remaining block (len(results)-from+1 entries) on every parallel pass, on the calling goroutine — i.e. inside the serial barrier this PR is trying to shrink.
The scan in firstUnacceptedResult is self-limiting (workers bail once stop drops below their index), so a pass typically advances only a short distance, but the prefix sum is paid in full regardless. With one conflict roughly every occMinParallelValidation+ transactions the backoff never engages, so you get ~C passes each doing O(N) serial work: O(C·N) total. At N=100k with C≈1k that is ~100M writes and ~800MB of allocation churn per block, all serial.
Capping the pass window (to = min(to, from+window)) would bound both the prefix sum and the atomic stop.Load() contention, at the cost of an extra pass on wide conflict-free runs.
There was a problem hiding this comment.
Not changed here by design: this stack re-opens the already-merged code unmodified so the team can review what actually runs on giga-1 (the stack top equals current giga-1). Noted as a follow-up.
a004fe1 to
a96c76d
Compare
bdchatham
changed the title
a96c76d to
971643c
Compare
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I tested this PR at commit |
|
Superseded by #4457, a backport of this change (#4261) to main. This stack targeted stacked giga-1 branches and could not merge to main; #4457 is based on main, resolves the conflicts with main's renames, and adds two review fixes (merge fragments are cleared before returning to the pool, and each parallel pass's look-ahead is bounded). |
… acceptance barrier (sei-protocol#4457) Backport: this ports sei-protocol#4273, the reviewable version of sei-protocol#4261, to main from the giga-1 PR stack sei-protocol#4270–sei-protocol#4273. The stack targeted stacked giga-1 branches and never reached main; only its first PR landed on main, as sei-protocol#4366. @shemnon approved sei-protocol#4273 at 1143167. Since that commit, the changes are main's renames (`baseAccount`, the `phaseOCC*` constants, `takeAccessSets`) and two review fixes: merge fragments are cleared before they return to the pool, and each parallel pass's look-ahead is bounded. On giga-testnet-2, `occ_validate` and `occ_merge` took about a third of the executor main loop, all of it on the calling goroutine. Most of that time went to transactions that the frontier accepts as they stand. A block of about 1,800 transactions holds less than one conflict. On `giga-1`, sei-protocol#4261 moved this work onto the OCC pool. It cut validate and merge from about 8.5 ms to 4.1 ms per block. That gave about 20% more blocks per second during the 95k tx/s window. sei-protocol#4269 reverted it for the reviewable stack, and that stack (sei-protocol#4273) never reached main. This PR ports sei-protocol#4261 and the sei-protocol#4273 follow-up to main (PLT-1377). Acceptance stays a serial barrier in block order. `stateAccessIndex` and `blockSTMState` split into 64 address shards. `indexResults` records the writes of every speculative result up front, and `conflictsWithin(key, sourcePrefix, txIndex)` replaces `conflictsWithAfter`. `acceptValidatedPrefix` finds the first result that the serial frontier would not accept, and `applyRange` folds the run before it into the prefix shard by shard. `validateBlockSTMFrontier` then handles only that one result, and `serialBackoff` keeps dependency chains on the calling goroutine. Each pass looks ahead at most twice as far as the previous pass accepted, and at least 2,048 results, so the work of a block's passes stays linear in its size. `changeSetIntoParallel` builds the changeset of each shard on the pool and joins the shards in address order. Its per-shard base-row cache replaces `prefetchBaseAccounts`. The port needs nothing from sei-protocol#4258 or sei-protocol#4260; the only conflicts came from the `baseAccount` and phase-constant renames on main. Non-app-hash-breaking: changesets, receipts, and tx results stay byte-identical to main. Shards are contiguous ranges of the first address byte, so shard order is canonical address order. Each address lives in one shard, so the apply order in a shard is block order. The new index can only add conflicts, from stale writes of an earlier incarnation. An extra rerun executes against the exact accepted prefix, so it gives the sequential result, and the extra reruns show only in OCC metrics. Review `stmFrontierAccepts` most closely, because it must mirror `needsSTMRerun`, and `touchedShards`, because it must cover every address that `applyOwned` and `indexResults` touch. `TestOCCRandomizedConflictingBlocksMatchSequential` checks seeded dense and sparse blocks at 3 and 8 workers against the sequential executor, down to the encoded FlatKV pairs. A seeded digest run gave identical output on main and this branch for 864 blocks, including 3,000- and 6,000-transaction blocks that cross a pass's look-ahead. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
3 participants
Describe your changes and provide context
Re-opens #4261 on top of #4272 (stack 4/4). With this PR applied the tree is byte-identical to today's
giga-1.OCC acceptance stays a serial barrier in block order, but the work behind it is spread across the existing worker pool (
occ_shards.go):stateAccessIndex);conflictsWithin) and reports the first one the frontier would not accept (firstUnacceptedResult); the frontier handles only that tx on the calling goroutine, then reruns from there (with aserialBackoffto avoid thrashing on hot contracts);occShardOf: contiguous address ranges), each incarnation recording which shards it touched so workers skip results holding nothing of theirs;mergeOCCResultsemits the changeset one shard at a time on the pool and concatenates shards in canonical address order, so the output is identical to the serial merge; a prefix with few keys is merged on the calling goroutine.Determinism: acceptance decisions and the emitted changeset/receipts are the same as the serial path by construction (parity tests compare both).
Testing performed to validate your change
scripts/ramtest.sh -race ./giga/evmonly/...— parity vs sequential validation/merge,conflictsWithinbounds,occShardOf,cumulativeGasFrom,firstUnacceptedResult, boundary rejection atresults[0].make fmtcheck,make lint.Link to Devin session: https://app.devin.ai/sessions/ff612badcded4aa5914ea408dbb41888
Open in Devin Desktop: https://app.devin.ai/desktop/session/ff612badcded4aa5914ea408dbb41888?variant=devin
Requested by: @bdchatham