feat(autobahn): order producer admission by shard ownership, then priority

[bdchatham]

The Autobahn producer mempool admits blocked InsertTx calls in arrival order and ignores fees, which the TODO(gprusak) on InsertTx called out for the case where a node takes more inserts per second than its lane can carry. That case is now routine under saturation (PLT-1366): a full lane gives its capacity to whoever arrived first, and non-owned transactions that the router keeps locally when their owner is not connected compete with the lane's own shard. This builds on #4451 and is tracked as PLT-1374.

Waiting inserts are now ranked by shard ownership, then priority, then arrival. The EvmOnly app reports the effective gas price as Priority from CheckTx, and GetTxPriorityHint computes the same value without sender recovery, so poolCanAdmit can still reject before CheckTx when a newcomer does not outrank the worst waiter. The hint holds a CheckTx permit, because an app may recover the sender to compute it. Ownership comes from the same commit epoch the router shards by (Committee.EvmShard for the session lane's validator), and it ranks admission only when Config.RankByShardOwnership is set, which the node sets from enable_evm_proxy: without the proxy a non-owned sender has no other lane to reach. When the key is not in the committee every transaction counts as owned. The FIFO waiters slice is replaced in admission.go by one google/btree ordered by rank, whose Min is the call to wake and Max the eviction candidate, plus a nonce-ordered queue per EVM sender, so only a sender's next nonce competes and a higher fee never passes an earlier nonce. At the MaxPendingInserts bound a better call evicts the worst one, which returns the retryable errPendingFull and is counted by a new producer_evictions metric. The evicted call is the last queued call of the worst waiter's sender rather than the waiter itself, because evicting the head would leave that sender's later nonces next in line to fail with errBadNonce. A call evicts only when its sender's queued nonces, with it added, form a gap-free run from the sender's next nonce: the mempool's tracked nonce, or the app nonce read before CheckTx if no block was pruned since. Any other call queues while there is room and gets errPendingFull at the bound, and a call whose next nonce is unknown never evicts. A call that repeats a nonce already queued for its sender, or whose nonce is below the sender's next nonce, fails with errBadNonce without queueing, so there is no replace-by-fee: a same-nonce resubmission, such as a fee bump, fails while the original is queued. With no priority hint and unknown ownership the behaviour equals #4451, and the existing producer tests pass unmodified.

This changes admission policy that clients can observe but not consensus or state, so it is non-app-hash-breaking. One design point wants a reviewer's call. The pre-CheckTx check cannot know the sender without sender recovery, so it assumes owned and cannot see a duplicate, stale, gapped or far-future nonce (those are rejected after CheckTx and never evict); when the worst waiter is a fallback transaction, every new insert pays for CheckTx, which weakens #4451 under a flood of non-owned traffic (still bounded by MaxConcurrentCheckTx). The clean fix is for the RPC layer to pass the sender it already recovers for routing, which changes the InsertTx interface. Ownership is fixed when a call joins the queue, there is no aging, so with the proxy on a fallback call can wait or be evicted while owned load keeps arriving, and Proxy.GetTxPriorityHint does not recover from panics the way CheckTxSafe does. New tests in admission_test.go cover rank order, owned before fallback, eviction and pre-CheckTx rejection, per-sender nonce order, equal ranks that never replace one another in the tree, duplicate, stale, gapped and far-future nonces that never evict, a model that checks every gap-free run against the expected nonce, ownership ignored without the flag, the hint waiting for a CheckTx permit, close, a model-based queue test and a stress test with evictions, all under -race.

[bdchatham]

@seidroid review

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Oct 5, 2026, 5:25 PM

[cursor]

PR Summary

Medium Risk
Changes observable mempool admission and eviction behavior under saturation (nonce ordering, fee bumps, proxy fallback traffic) without consensus/state changes; complexity and concurrency in the new queue increase regression risk.

Overview
Replaces FIFO blocked InsertTx waiting with a ranked admission queue in admission.go: optional shard ownership (when RankByShardOwnership / EVM proxy is on), then CheckTx priority, then arrival. Only each EVM sender’s lowest queued nonce competes globally; later nonces wait behind it.

At MaxPendingInserts, a higher-ranked insert can evict the tail of the worst eligible sender’s queue (only when its nonce keeps a gap-free run from the expected next nonce). Duplicate, stale, and gapped nonces get errBadNonce or errPendingFull without replace-by-fee. GetTxPriorityHint plus EvmOnly Priority (effective gas price) enable a pre-CheckTx reject when the queue is full; hints use a CheckTx permit. Adds producer_evictions metrics and broad tests in admission_test.go.

^{Reviewed by Cursor Bugbot for commit c5c1fa9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 91.42857% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.78%. Comparing base (880848f) to head (c5c1fa9).

Files with missing lines	Patch %	Lines
.../internal/autobahn/producer/metrics/metrics.gen.go	0.00%	9 Missing ⚠️
...tendermint/internal/autobahn/producer/admission.go	96.58%	4 Missing ⚠️
...i-tendermint/internal/autobahn/producer/mempool.go	95.45%	3 Missing ⚠️
...mint/internal/autobahn/producer/metrics/metrics.go	0.00%	1 Missing ⚠️
sei-tendermint/internal/evmonlyapp/app.go	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4452      +/-   ##
==========================================
+ Coverage   56.77%   56.78%   +0.01%     
==========================================
  Files        2126     2127       +1     
  Lines      166876   167050     +174     
==========================================
+ Hits        94737    94861     +124     
- Misses      72134    72184      +50     
  Partials        5        5              

Flag	Coverage Δ
sei-chain	54.96% <91.42%> (+0.02%)	⬆️
sei-db	75.17% <ø> (ø)
sei-db-state-db	78.82% <ø> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
sei-tendermint/internal/autobahn/producer/state.go	84.84% <100.00%> (+0.15%)	⬆️
sei-tendermint/node/setup.go	66.82% <ø> (ø)
...mint/internal/autobahn/producer/metrics/metrics.go	0.00% <0.00%> (ø)
sei-tendermint/internal/evmonlyapp/app.go	87.66% <88.88%> (+0.02%)	⬆️
...i-tendermint/internal/autobahn/producer/mempool.go	94.50% <95.45%> (+0.82%)	⬆️
...tendermint/internal/autobahn/producer/admission.go	96.58% <96.58%> (ø)
.../internal/autobahn/producer/metrics/metrics.gen.go	0.00% <0.00%> (ø)

... and 26 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

This replaces the FIFO admission queue for blocked InsertTx calls with a ranked queue: senders of a shard this validator owns go first, then higher priority, then earlier arrival. Calls from one sender stay in nonce order, the worst call is evicted when the queue is full, and a priority hint lets a call that would lose be rejected before CheckTx. Against the PR's merge tree (8e3a455), the heap, nonce-chain and eviction bookkeeping and the wakeup paths hold up, so nothing blocks; the notes below are about cost and a doc mismatch, and the only other reader (codex) returned nothing, so there were no outside findings to keep or drop.

Non-blocking

1 finding on the changed lines, as inline comments.

• Ownership is the first sort key and calls never age, so under steady saturation fallback (non-owned) inserts can wait forever and then be evicted, however much they pay. When EnableEvmProxy is false these inserts have no other route to a lane, so for operators who turn the proxy off, roughly (n-1)/n of senders are penalised by address hash alone. The description already flags this. It is worth gating the ownership key on the proxy flag, or adding a bounded aging term, before this ships to configurations with the proxy off.

1 nit, not posted on the code

• sei-tendermint/internal/autobahn/producer/mempool.go:233 — The doc says an evicted or outranked call fails with errMempoolFull, but enqueue and the evicted check both return errPendingFull.

_{seidroid review · decision approve · session d024a7480b524ef1a01f7db4e8c8edea · turn resp_claude_82a82dbdc81452da18695bb1a082217a · item aeb68d22440a50d195c8a9f02ce1881b}

Findings: 0 blocking | 2 non-blocking | 1 posted inline

[bdchatham]

@seidroid review

[seidroid]

Since the last review, the pre-CheckTx priority hint holds a CheckTx permit, ownership only affects ranking when RankByShardOwnership is set (the node takes it from enable_evm_proxy), and the errPendingFull doc is corrected; I checked each against the diff and all three earlier points are addressed. Nothing blocks: I kept codex's one finding as a suggestion, because I confirmed that a same-nonce or gapped call can evict a usable one, but it gets no further than the free queue-filling the old FIFO already allowed; codex was the only reader and raised nothing else.

Non-blocking

1 finding on the changed lines, as inline comments.

_{seidroid review · decision approve · session d024a7480b524ef1a01f7db4e8c8edea · turn resp_claude_f481e5420d248722696df4f9b8880cc4 · item e508a4947d49519ebbe06d758b3e443e}

Findings: 0 blocking | 1 non-blocking | 1 posted inline

[bdchatham]

For metrics, can we use the otel telemetry setup instead of this legacy metric generation? Let's not do this if it adds a bunch of scope but please create a linear ticket for me to do this in that case

[bdchatham]

Tracked in PLT-1375 (https://linear.app/seilabs/issue/PLT-1375). Moving only this counter would split the producer package across two metric systems. All five Autobahn metrics packages use metricsgen, and the giga-testnet alerts and dashboards read their series names, so the move is a package-by-package change that keeps every name, label and bucket. internal/mempool/metrics.go is the OTel pattern to follow.

[bdchatham]

Constants should be at the top of files please.

[bdchatham]

Done in 3b3327b. Package-level consts now sit directly after the imports, with their doc comments, in every file this PR changes: admission.go, state.go and evmonlyapp/app.go.

[bdchatham]

@seidroid review

[seidroid]

Since the last review, a call that repeats a nonce already queued for its sender is rejected with errBadNonce, and eviction is limited to calls that Fit classifies as contiguous. I confirmed the duplicate half of my earlier finding is fixed, but codex's one new finding holds and I kept it: Fit only compares against the ends of the queue, never the sender's expected next nonce, so a call that is certain to fail can still evict. That matches the weight of my earlier note, so it stays a suggestion and the verdict stays approve; codex was the only reader and raised nothing else.

Non-blocking

1 finding on the changed lines, as inline comments.

_{seidroid review · decision approve · session d024a7480b524ef1a01f7db4e8c8edea · turn resp_claude_e38f05681753a1e8cd5c82664bfb0376 · item 5c4364a45b59528a99e416f87c95f039}

Findings: 0 blocking | 1 non-blocking | 1 posted inline

[seidroid]

review found nothing blocking in the commit it read. It does not approve: this pull request was pushed to while the review ran, so the approval would stand against code the review never saw. Ask for another review to have the new commit read.

[bdchatham]

@seidroid review

[seidroid]

Since the last review, the two heaps of eligible calls were replaced by one google/btree ordered by rank. Every queued call gets a unique seq, so no two compare equal, and the module already depends directly on google/btree. The Fit contiguity check is unchanged, so my open thread on it still holds. Codex raised the same point at the moved line: I kept it and report it once, replacing that thread; codex added nothing else, and nothing blocks.

Non-blocking

1 finding on the changed lines, as inline comments.

_{seidroid review · decision approve · session d024a7480b524ef1a01f7db4e8c8edea · turn resp_claude_efa319459ee048579b040516b483f733 · item 1becef6fa6f95a539f018d2d416bb824}

Findings: 0 blocking | 1 non-blocking | 1 posted inline

[bdchatham]

@seidroid review

[seidroid]

Since the last review, Fit checks contiguity against the sender's expected next nonce, which comes from mempoolInner.expectedNonce, the same tracked-or-pre-read rule appendTx now shares. Only a gap-free run from that nonce may evict, a call below it is rejected with errBadNonce before it queues, and an unknown expected nonce never evicts. That addresses my last open finding, so nothing blocks; codex found nothing, and I found nothing new either.

_{seidroid review · decision approve · session d024a7480b524ef1a01f7db4e8c8edea · turn resp_claude_05cb548fd85d1eb189a7fef4db5e6660 · item 9757faaffcdd53dd836f023ac67c549e}

Findings: 0 blocking | 0 non-blocking | 0 posted inline