Repository navigation
PLT-1072: Wire rate limiter into native gRPC (:9090) unary+stream interceptors - #4021
Conversation
Applied per-IP token-bucket admission on native gRPC
before handlers run, emitting rpc_rate_limit_rejected_total{plane="grpc"}.
Co-authored-by: Cursor <cursoragent@cursor.com>
amir-deris
changed the title
PR SummaryMedium Risk Overview Native gRPC (:9090) uses a tap handler so throttling happens before protobuf decode; unary/stream interceptors cover streams (one token to open, one per inbound message) and avoid double-charging when the tap or gRPC-Web middleware already admitted the RPC. gRPC-Web (:9091) uses HTTP middleware (OPTIONS skipped) and shares the same registry as :9090. Rejections are Config and ops: Reviewed by Cursor Bugbot for commit 8680af6. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![seidroid[bot]](https://avatars.githubusercontent.com/in/2745135?s=60&v=4){kind=small}
There was a problem hiding this comment.
The config plumbing, metrics bucketing, and characterization-suite updates are solid, but the unary rate-limit interceptor is effectively dead code: BaseApp's RegisterGRPCServer wrapper discards the interceptor grpc-go hands to each MethodDesc.Handler, so no module Query service on :9090 is actually rate limited.
Findings: 1 blocking | 2 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
- 1 blocking issue(s) flagged inline on specific lines.
Non-blocking
- [suggestion] All interceptor tests invoke the returned
grpc.UnaryServerInterceptor/StreamServerInterceptorclosures directly, so none of them exercise grpc-go's dispatch path. That is exactly why the BaseApp bypass is invisible to the suite. Add a test that starts a realgrpc.ServerviaStartGRPCServer(or at minimum registers a service throughBaseApp.RegisterGRPCServer), dials it, and assertscodes.ResourceExhaustedafter the burst is spent. - 1 suggestion(s)/nit(s) flagged inline on specific lines.
BaseApp.RegisterGRPCServer rebuilt every grpc.MethodDesc with a handler that discarded the grpc.UnaryServerInterceptor grpc-go passes in. Since grpc-go hands s.opts.unaryInt to MethodDesc.Handler rather than invoking it itself, every service registered through RegisterGRPCServer -- all module Query services, cosmos.tx.v1beta1.Service and cosmos.base.tendermint.v1beta1.Service -- ran no server-level interceptor. The rate limiter wired onto :9090 therefore covered only the directly-registered reflection service. Thread the incoming interceptor into the chain at that wrapper handler, the single point every unary query passes through, so any grpc.ChainUnaryInterceptor applies without each registration site re-adding it. The query-context interceptor runs last, so admission control rejects a call before it costs a query context. Add tests that register through RegisterGRPCServer on a real grpc.Server and dial it, covering both the generic interceptor contract and the rate limiter reaching ResourceExhausted once a burst is spent. The existing interceptor tests invoked the closures directly, which is why the bypass was invisible. Also correct the stale [grpc] key counts and clamp line reference in the configuration characterization suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@seidroid review |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
There was a problem hiding this comment.
The interceptor wiring and the BaseApp dispatch fix from the previous round look correct and are now covered by real-server regression tests. However, the four new [grpc] config fields are auto-derived into the config/registry key space, and config/cosmosbase was not updated to match, which breaks two characterization tests on the base branch.
Findings: 1 blocking | 5 non-blocking | 5 posted inline
Blockers
config/cosmosbasewas not updated for the four new[grpc]keys, and two of its tests will fail.registry.RegisterSection(GRPCSectionName, &srvconfig.GRPCConfig{}, grpcDefaults)(config/cosmosbase/cosmosbase.go:40) derives the section's declared keys fromGRPCConfig'smapstructuretags (config/registry/registry.go:144→deriveKeys/walk), so addingip-rate-limit-rps,ip-rate-limit-burst,rate-limiting-enabled, andtrusted-proxy-cidrstakes the registered set from 11 to 15 keys automatically. Two tests still hold the old set: (1)TestTheGRPCKeysAreTheOnesItsReaderResolves(config/cosmosbase/cosmosbase_test.go:75-81) doesreflect.DeepEqualagainst a hard-coded 11-key list and will fail; (2)TestEveryKeyTheseSectionsDeclareIsOneTheReaderResolves(config/cosmosbase/agreement_test.go:184) walks every registered key and errors for each one absent fromreaderValues, which still stops atgrpc.keepalive-permit-without-stream(agreement_test.go:102-112) — four errors. Fix: add the four keys to therequireDeclareslist and add the fourcfg.GRPC.*entries toreaderValues. While there, thegrpcDefaultsgodoc (cosmosbase.go:99) still reads "Six of these eleven keys are read only when the key is present"; it is now nine of fifteen, and per AGENTS.md that count is part of the record rather than incidental prose.
Non-blocking
- None at the file/PR level.
- 5 suggestion(s)/nit(s) flagged inline on specific lines.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4021 +/- ##
==========================================
- Coverage 61.23% 60.24% -0.99%
==========================================
Files 2177 2056 -121
Lines 190653 176242 -14411
==========================================
- Hits 116749 106184 -10565
+ Misses 62894 60333 -2561
+ Partials 11010 9725 -1285
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Register the four new [grpc] keys in config/cosmosbase so characterization tests stay in sync, fix config-test doc/grouping drift, and correct knownGRPCServices for reflection metrics. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@seidroid review |
There was a problem hiding this comment.
The previous round's blocker is genuinely fixed: the BaseApp dispatch bypass is closed at the one wrapper handler every unary query passes through, real-server regression tests cover it, and the four new [grpc] keys are now declared in config/cosmosbase with consistent counts and a corrected knownGRPCServices list. Two suggestions from the last review remain open (no operator signal when admission is enabled with a zeroed bucket, and the unstated per-stream admission scope), and gRPC-Web silently shares these interceptors despite being listed as out of scope.
Findings: 0 blocking | 5 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- [suggestion] gRPC-Web is not actually out of scope:
StartGRPCWebwraps this same*grpc.Server(sei-cosmos/server/grpc/grpc_web.go:29), so both interceptors also govern:9091, and its traffic draws tokens from the same per-IP buckets as:9090. grpc-go's handler-server transport does set the peer fromreq.RemoteAddr, so per-IP attribution works there, but it means[grpc] trusted-proxy-cidrsalso decides whosex-forwarded-foris honoured on the browser-facing port. Worth confirming that is intended and saying so on the config keys rather than leaving the PR description's "Out of scope: gRPC-Web" to imply :9091 is untouched. - [suggestion] Nothing exercises
StartGRPCServer's enable path.rate_limit_dispatch_test.gobuilds its owngrpc.NewServer(grpc.ChainUnaryInterceptor(...)), and no test callsStartGRPCServerwithRateLimitingEnabled = true, so deleting the twoserverOptsappends — or theratelimiter.Newerror return for a malformed CIDR — leaves the suite green. That is the same class of gap that hid last round's BaseApp bypass, one layer up. - [suggestion] The three
v.IsSet-guarded keys are recorded only by name and for their absent-key resolution; no test reads a value that is actually present. The six pre-existing guarded[grpc]keys are covered on that path byTestGetConfigGRPCOverrides(config_test.go:167-190), which was not extended. A typo repeated in both theIsSetcall and the getter (e.g.grpc.ip-rate-limit-rpin both) would pass every test in the suite today. Addingip-rate-limit-rps,ip-rate-limit-burstandtrusted-proxy-cidrsto that test closes it. - 2 suggestion(s)/nit(s) flagged inline on specific lines.
Superseded: latest AI review found no blocking issues.
- Log at startup when admission is enabled with a zeroed token bucket, matching the signal the CometBFT RPC plane already emits. - State the per-stream (not per-message) admission scope in the stream interceptor godoc. - Document that the interceptors and trusted-proxy-cidrs also govern gRPC-Web (:9091), which wraps the same server. - Cover StartGRPCServer's enable path, disable path, and malformed-CIDR failure against a real server. - Extend TestGetConfigGRPCOverrides with the three IsSet-guarded [grpc] rate-limit keys. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Streams now cost one token to establish and one per inbound message, so an admitted reflection stream can no longer serve unbounded descriptor lookups. PlaneGRPC rejection metrics record the full "service/Method" instead of the service name, bounded by the methods actually registered on the server (read from GetServiceInfo at startup) rather than a hand-maintained allowlist. Unregistered methods still record "other". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@seidroid review |
|
@seidroid review |
| // srv. | ||
| func registeredMethods(srv *grpc.Server) []string { | ||
| info := srv.GetServiceInfo() | ||
| methods := make([]string, 0, len(info)) |
There was a problem hiding this comment.
why prealloc by service info? we know the exact size of this slice.
| RPS: c.IPRateLimitRPS, | ||
| Burst: c.IPRateLimitBurst, | ||
| TrustedProxyCIDRs: c.TrustedProxyCIDRs, | ||
| } |
There was a problem hiding this comment.
Small note that I think this would end up using a single bucket for all loopback addresses. This is OK for now. Worth documenting specifically.
There was a problem hiding this comment.
Thanks for feedback. Will add documentation.
There was a problem hiding this comment.
The gRPC plane wiring is sound: the enable path, the disabled path, and the malformed-CIDR failure are now covered by real-server tests, the zeroed-bucket warning sits in the one function every gRPC start passes through, per-message stream charging closes the unbounded-reflection gap, and the metric label set is derived from the server's own registered methods rather than a hand-maintained list — every open item from the previous rounds is addressed. Remaining findings are record-accuracy issues in the configuration characterization suite plus two stray empty files.
Findings: 0 blocking | 4 non-blocking | 2 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- [suggestion] Two zero-byte files are added under
sei-cosmos/server/config/testdata/—grpc.keys.goldenandserver_config.golden— and nothing in the tree references either path. There is no golden/writeGoldeninfrastructure insei-cosmos/server/configat all (the only golden helpers live intestutil/configtestandcmd/seid/cmd), so these are empty records no test can ever compare against or fail on. They read as leftovers from an earlier iteration of the branch; either delete them or add the test that reads them. - 2 suggestion(s)/nit(s) flagged inline on specific lines.
- 1 non-blocking pre-existing issue(s) listed below under pre-existing issues.
Pre-existing issues
- [suggestion]
sei-cosmos/server/config/config_fuzz_test.go:764citesconfig.go:519-521as saying why the gRPC guards matter ("a node upgrading with an older app.toml stays bounded"), but that range holds thestate-commit.flatkv.*presence guards, not anything gRPC. The reference was already wrong on the base branch (it pointed atstate-commit.sc-snapshot-*there); the bounded-gRPC guards live atconfig.go:581-587.
gRPC decodes the request message before the interceptor chain runs, so the unary interceptor could shed a throttled caller's handler work but never the decoder's. Methods with expensive decodes, such as GetTxsEvent, were still reachable by a caller over its budget. - Add RateLimitTapHandle, a grpc.InTapHandle that charges the per-IP bucket when a stream's headers arrive, before any message is read off the wire or unmarshalled. It admits native gRPC (:9090). - Add RateLimitHTTPMiddleware for gRPC-Web (:9091), which reaches the same server through ServeHTTP and so never runs the tap handler. Over-budget callers get HTTP 429 there; CORS preflight is exempt so a browser does not pay two tokens per call. - Record the admission on the request context, which becomes the stream context, so the interceptors do not charge an RPC twice. They now cover the per-message stream charge and any RPC that arrived uncharged. - Return the registry from StartGRPCServer and pass it to StartGRPCWeb. A registry built separately for gRPC-Web would give each IP a second bucket, doubling a client's budget across the two ports. The stream per-message charge stays after RecvMsg: charging before the read would throttle a caller for a message it never sent, killing an idle stream whose client is waiting rather than sending. At most one message per stream is decoded past the budget, bounded by MaxRecvMsgSize. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gRPC plane read its per-IP defaults from ratelimiter.DefaultRPS and DefaultBurst, whose names claim a scope wider than the one plane that consumes them. Declare the numbers in the gRPC config package alongside the other DefaultGRPC* knobs, matching how the EVM and CometBFT planes already carry their own, and leave the ratelimiter constants at 200/400. The stream path charges a token per inbound message, so a burst of 20 is low enough to cut off a streaming client mid-call. Admission is still off by default; note the sizing consideration in the upgrade guide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8680af6. Configure here.
| ip := registry.IPFromHTTPRequest(r) | ||
| if !registry.Allow(r.Context(), ip, ratelimiter.PlaneGRPC, r.URL.Path) { | ||
| http.Error(w, "too many requests", http.StatusTooManyRequests) | ||
| return |
There was a problem hiding this comment.
gRPC-Web 429s omit CORS headers
Medium Severity
RateLimitHTTPMiddleware writes the 429 and returns before grpcweb.WrapServer, so EnableUnsafeCORS never attaches CORS headers to a rejected call. Browser clients then treat the rate-limit response as a CORS failure instead of HTTP 429.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 8680af6. Configure here.
…he gRPC query plane (sei-protocol#4078) Closes [PLT-1125](https://linear.app/seilabs/issue/PLT-1125/bound-concurrent-in-flight-rpcs-and-connections-per-ip-on-the-grpc). ## Impact Caps how much load a single client IP can place on the gRPC query plane (`:9090` and `:9091`): - **Connections:** optional per-IP limit via `max-connections-per-ip` (default **0**, unlimited). Set a positive value to cap one address's share of the global connection budget. - **In-flight RPCs:** when rate limiting is enabled, each IP is limited to 100 concurrent RPCs (`max-in-flight-per-ip`), complementing the per-IP token bucket from sei-protocol#4021. Both planes share the same per-IP pools. Rejections are counted separately from rate-limit rejections (`rpc_connection_rejected_total`, `rpc_inflight_rejected_total`). **Operators:** clients behind a shared egress or reaching the node over `127.0.0.1` share one allowance when a positive cap is set. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…4336) Replaces the hand-written `## Unreleased` section with the generated PR list, in the format every released version uses: the heading, `sei-chain`, and one line per PR. Generated with `./scripts/generate-changelog.sh release/v6.7 main` (175 PRs merged to `main` since `release/v6.7` branched), with the script's `## main` heading renamed to `## Unreleased`. The eight PRs that had hand-written entries (sei-protocol#4166, sei-protocol#4098, sei-protocol#4191, sei-protocol#4117, sei-protocol#4078, sei-protocol#4032, sei-protocol#4021, sei-protocol#4009) are all in the list. `main` only, so no backport label: `release/v6.7` has no Unreleased section, and cherry-picking this there conflicts. It is independent of sei-protocol#4335, which touches only `## v6.7`; the two merge cleanly in either order. Docs-only; no code change. Made with [Cursor](https://cursor.com) Co-authored-by: Cursor <cursoragent@cursor.com>
3 participants
Summary
Wires the shared
ratelimiter.Registry(from PLT-411 / PLT-800) into native gRPC on:9090via unary and stream server interceptors. Rejections returncodes.ResourceExhaustedand emitrpc_rate_limit_rejected_total{plane="grpc", method_namespace="..."}.This is the last Phase 1 plane; EVM HTTP (PLT-819), CometBFT HTTP (PLT-981), and the core registry/parser are already landed.
sei-cosmos/server/grpc/rate_limit.go) — per-IP token bucket at unary call and stream establishment; usesRegistry.IPFromGRPCContextandinfo.FullMethod(noMethodParser/ body pre-read needed on gRPC).StartGRPCServer) — interceptors chained at server creation, before BaseApp query handler registration, so admission runs beforesdk.Contextcreation.[grpc]inapp.toml) —rate_limiting_enabled,ip_rate_limit_rps,ip_rate_limit_burst,trusted_proxy_cidrs; ships disabled by default (same rollout pattern as 1b/1c).ratelimiter/method_bucket.go) — addsPlaneGRPCand low-cardinality service-name labels for known protobuf services.Out of scope: gRPC-Web (
:9091, Phase 4d).Test plan
go test ./ratelimiter/... ./sei-cosmos/server/grpc/... ./sei-cosmos/server/config/...ResourceExhausted, per-IP isolation, trusted-proxy XFF[grpc]keysgrpc.rate-limiting-enabled = trueon a dev node and confirm 429-equivalent gRPC rejections under burst load