perf(autobahn): reject an insert into a full producer mempool before CheckTx

[bdchatham]

An insert into a full producer mempool ran the app CheckTx before it learned that the mempool had no room. CheckTx does ecrecover and writes the sender cache, so each such insert paid the full CheckTx cost and then failed. On a trapped giga-testnet-0 validator, about 4.4x more CheckTx attempts ran than inserts succeeded (ok 797/s, pending_full 3,476/s). That wasted CPU keeps a slow validator behind under saturation load. Linear: PLT-1371 (from PLT-1366).

State.doInsertTx now calls poolCanAdmit after it gets the session mempool and before checkTx. That function takes the mempool lock once. It returns the error that the admission loop would return at that moment for an insert without a ticket. That is ErrNotProducing for a closed session, errMempoolFull for TryInsertTx when IsFull(), and errPendingFull for InsertTx when len(waiters) >= maxPendingInserts(). The errors are the same values, so insertTx counts them under the same producer_inserts{result} labels. For a valid tx, RPC callers see the same errors as before. The admission loop after CheckTx is unchanged and stays authoritative. The check also runs before the CheckTx permit wait, so a rejected insert does not hold a checkTxSem slot or a tryInsertSem slot for the CheckTx duration.

Two points differ from the ticket text. The pending check omits IsFull(): with no ticket and len(waiters) >= max >= 1, isHead is false, so the loop rejects with errPendingFull whether or not the mempool is full. The closed check comes first, because the loop also checks closed first; without it, a closed and full session would report full where the loop reports not_producing.

Error order. The order changes for a tx that is both invalid and sent to a full mempool. Before, it got its non-retryable CheckTx rejection, or errTooLarge from the gas checks. Now it gets the retryable errMempoolFull or errPendingFull. The client then retries a tx that can never land, and it learns that only when there is room. Under saturation this is the intended trade: CheckTx is the cost this change removes. TestTryInsertTx_FullRejectsInvalidTxAsFull pins the new order.

Race analysis. Each early rejection is exactly the loop's answer at the moment of the check. After the check, other inserts can fill the mempool, and a prune can free it by advancing m.first. Thus an insert that the check rejects could have succeeded only if capacity freed during its own permit wait and CheckTx. One permit wait plus one CheckTx bounds that window. In that case the caller gets the documented, retryable mempool-full error, the same answer as with an instantaneous CheckTx. Under saturation, I expect other in-flight inserts to take the freed capacity, so throughput does not drop.

A closed session never reopens, so the closed check has no race. The happy path takes one more short mempool lock (three instead of two). The admit and check_tx latency histograms get fewer samples, because early rejections skip both phases.

Tests: three new tests in checktx_limit_test.go use a countingApp. They show that TryInsertTx on a full mempool, InsertTx with MaxPendingInserts queued, and both calls on a closed session return the expected result and do not call CheckTx. All three fail on main. The existing producer tests pass with no change.

go test -race -count=1 ./sei-tendermint/internal/autobahn/producer/...   ok
go test -race -count=20 ./sei-tendermint/internal/autobahn/producer/     ok
gofmt -s -l / goimports -l (changed package)                              no output
go vet ./sei-tendermint/internal/autobahn/producer/...                    no output
staticcheck 2026.2.1 (0.8.1) ./sei-tendermint/internal/autobahn/producer/...   no output
golangci-lint v2.13.2 run ./sei-tendermint/internal/autobahn/producer/...      0 issues.
golangci-lint v2.13.2 fmt --diff ./sei-tendermint/internal/autobahn/producer/... no output

🤖 Generated with Claude Code

[bdchatham]

@seidroid review

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Oct 5, 2026, 3:25 PM

[cursor]

PR Summary

Medium Risk
Changes producer insert error ordering and hot-path mempool admission under load, though post-CheckTx admission stays authoritative and outcomes for valid txs match prior behavior.

Overview
Inserts that cannot be admitted now fail before running the app CheckTx, avoiding ecrecover and sender-cache work when the producer mempool is saturated.

doInsertTx calls new poolCanAdmit right after resolving the session mempool and before acquiring a CheckTx permit. Under one mempool lock it returns the same errors the post-CheckTx admission loop would use at that moment: ErrNotProducing (closed session), errMempoolFull for TryInsertTx when full, and errPendingFull when the pending-insert queue is at MaxPendingInserts. Post-CheckTx admission is unchanged.

Behavior change: a tx that is invalid and sent while the pool is full now gets a retryable full/pending-full error instead of an immediate CheckTx rejection; tests pin that ordering.

New countingApp tests assert CheckTx is not invoked for full, pending-full, and closed-session paths.

^{Reviewed by Cursor Bugbot for commit 66ba3ee. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.75%. Comparing base (137bede) to head (66ba3ee).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4451   +/-   ##
=======================================
  Coverage   56.75%   56.75%           
=======================================
  Files        2126     2126           
  Lines      166863   166871    +8     
=======================================
+ Hits        94704    94714   +10     
+ Misses      72154    72152    -2     
  Partials        5        5           

Flag	Coverage Δ
sei-chain	54.93% <100.00%> (+<0.01%)	⬆️
sei-db	75.17% <ø> (ø)
sei-db-state-db	78.84% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...i-tendermint/internal/autobahn/producer/mempool.go	93.67% <100.00%> (-0.61%)	⬇️

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[seidroid]

This change adds rejectBeforeCheckTx, which turns away an insert into a closed session, a full mempool (TryInsertTx) or a full pending queue (InsertTx) before the CheckTx permit wait and CheckTx itself. I checked each early rejection against the admission loop with no ticket, given maxPendingInserts() >= 1, and each returns the same error the loop would, so nothing blocks; codex's reading found nothing, which matches this one except for the error-priority note below. I reviewed against the PR's merge tree, but could not build or run the tests because the sandbox has no Go toolchain.

Non-blocking

1 finding on the changed lines, as inline comments.

_{seidroid review · decision approve · session db87b35dc278425eb175590acb20293b · turn resp_claude_50ba2e4ec87c2d4b55d1214833f9c6d8 · item e7f6622c9f225ac3a07201f0af3f9a5a}

Findings: 0 blocking | 1 non-blocking | 1 posted inline

[bdchatham]

@seidroid review

[seidroid]

Since my last review only the tests changed: TestTryInsertTx_FullRejectsInvalidTxAsFull now checks that an undecodable tx sent to a full mempool gets errMempoolFull without CheckTx running. The PR description now says the errors are unchanged only for a valid tx and explains the new error order, so my one suggestion (already marked resolved) is addressed, mempool.go is unchanged and still correct, and nothing blocks. codex again found nothing, which matches this reading, and I reviewed against the PR's merge tree but still could not run the tests because there is no Go toolchain here.

_{seidroid review · decision approve · session db87b35dc278425eb175590acb20293b · turn resp_claude_64c5b63f473fbe43667c434185d4198c · item 17927a3f751a5438b9c010219536e73f}

Findings: 0 blocking | 0 non-blocking | 0 posted inline

[bdchatham]

@seidroid review

[seidroid]

Since my last review the only change renames rejectBeforeCheckTx to poolCanAdmit and rewrites its doc comment. No old-name references are left in the tree, the logic and tests are unchanged, and my earlier error-order suggestion stays addressed, so nothing blocks. codex again found nothing, which matches this reading; I reviewed against the PR's merge tree but could not run the tests because there is no Go toolchain here.

_{seidroid review · decision approve · session db87b35dc278425eb175590acb20293b · turn resp_claude_3b2bd5f0df14e8a3993e98ffbf683bae · item 04d8fd6ee17a5ca1a749447b005be5fb}

Findings: 0 blocking | 0 non-blocking | 0 posted inline

[shemnon]

The unit tests only check the failure modes, are the existing positive tests sufficient with this new gate?

Otherwise, LGTM

[bdchatham]

@shemnon awesome, thanks for taking a look. Triple checked and the existing positive checks will cover this well