PLT-1072: Wire rate limiter into native gRPC (:9090) unary+stream interceptors

CHANGELOG.md

@@ -36,6 +36,7 @@ Ref: https://keepachangelog.com/en/1.0.0/
 * [#3990](https://github.com/sei-protocol/sei-chain/pull/3990) Freeze mode is limited to full nodes and disables transaction and evidence submission, mempool gossip, and state sync from startup while preserving query RPC and mempool-backed reads. Frozen and Autobahn nodes no longer advertise the unused mempool P2P channel.
 * [#4009](https://github.com/sei-protocol/sei-chain/pull/4009) Bound `/store/*/subspace` ABCI queries with pair/byte caps, empty-prefix rejection, SS-path concurrency limits, and context-aware iteration to prevent memory-exhaustion DoS.
 * [#4032](https://github.com/sei-protocol/sei-chain/pull/4032) fix(config): the default `telemetry.prometheus-retention-time` drops from `7200` to `0`, so neither app.toml-generation pipeline (`seid init`, or the file a node writes for itself on any other subcommand) starts the Prometheus metrics sink unless an operator sets a positive retention. Freshly generated nodes keep the bounded in-memory telemetry sink used by SIGUSR1 dumps. Existing `app.toml` files are unchanged.
+* [#4021](https://github.com/sei-protocol/sei-chain/pull/4021) feat(grpc): per-IP rate-limit admission for the gRPC plane, off by default behind `[grpc] rate-limiting-enabled` (new `ip-rate-limit-rps` / `ip-rate-limit-burst` / `trusted-proxy-cidrs`, defaults 10 rps / 20 burst / trust no proxy). Native gRPC (:9090) is admitted by a tap handler and gRPC-Web (:9091) by HTTP middleware, both before the request is protobuf-decoded, so a throttled caller cannot spend the decoder; streams pay one token to establish and one per inbound message. Both planes draw from the same per-IP buckets. Over-budget callers get `ResourceExhausted` on :9090 and HTTP 429 on :9091, counted by `rpc_rate_limit_rejected_total{plane="grpc", method_namespace}`.
 
 ### Upgrade guide
 * **IBC core removal.** Removes the retired IBC core source, protobufs, light clients, CLI, and simulation support. Retired IBC stores remain mounted but are omitted from `export-genesis`; preserve the state database or use v6.6 freeze nodes for historical IBC data.
@@ -46,6 +47,7 @@ Ref: https://keepachangelog.com/en/1.0.0/
 * **WebSocket frame size default drops from 10 MiB to 5 MiB.** Before this release, :8546 used a hardcoded 10 MiB frame cap. Both HTTP and WebSocket now share `[evm].max_request_body_bytes`, whose default is 5 MiB (`5242880`). WS clients that send frames in the 5-10 MiB range (large `eth_sendRawTransaction` batches, wide filter payloads, etc.) will be disconnected after upgrade unless the limit is raised. **Operators who relied on the old 10 MiB WS cap should set `max_request_body_bytes = 10485760` in `app.toml` before upgrading.** This also raises the HTTP body limit to 10 MiB. The exported `DefaultWebsocketMaxMessageSize` constant was removed; use the config knob instead.
 * [#4032](https://github.com/sei-protocol/sei-chain/pull/4032) **The Prometheus telemetry sink is off by default.** A node whose `app.toml` is generated by this release gets `prometheus-retention-time = 0`, which leaves the sink uncreated even though `telemetry.enabled` stays `true`. `GET /metrics?format=prometheus` on the app API server (:1317) then returns `prometheus metrics are not enabled`, and the `seid` process exports no application Prometheus series. **Operators who scrape application metrics should set a positive `[telemetry] prometheus-retention-time` in `app.toml` (the previous default was `7200`) before generating a new configuration file.** Nodes that already have `prometheus-retention-time` written in `app.toml` are unaffected.
 * [#3984](https://github.com/sei-protocol/sei-chain/pull/3984) **ABCI/gRPC pagination is now capped by default.** Untrusted callers requesting `limit` above 1000, `offset` above 10000, or a scan that exceeds 11000 total iterations now get `InvalidArgument` (over-cap) or a partial page with `next_key` (budget exhausted) instead of the previously unbounded scan. Clients that page with large limits/offsets, or trusted internal indexers, should either follow `next_key` for resumption or be added to the new `[query] trusted-cidrs` allowlist (or set `[query] disable-limits = true`) before upgrading.
+* [#4021](https://github.com/sei-protocol/sei-chain/pull/4021) **gRPC per-IP rate limiting defaults are deliberately conservative.** Admission stays off unless `[grpc] rate-limiting-enabled = true`, but the defaults it enables are 10 rps / 20 burst per IP. Streams pay one token to establish and one per inbound message, so at the default burst a client that sends more than ~20 messages in a burst is cut off mid-stream with `ResourceExhausted`. Operators enabling admission should size `ip-rate-limit-rps` / `ip-rate-limit-burst` against their heaviest streaming client, and set `trusted-proxy-cidrs` to their ingress CIDRs so callers are not all bucketed under the proxy's IP.
 * [#4009](https://github.com/sei-protocol/sei-chain/pull/4009) **`/store/*/subspace` scans are now capped.** Wide prefix scans that previously returned unbounded KV pairs now fail with `subspace result exceeds limit` once they would exceed the default caps of 1,000 pairs or 4 MiB of accumulated key+value bytes. Empty prefixes are rejected. Indexers and tooling that issue wide `/subspace` queries must narrow their prefixes, shard by sub-prefix, or raise `[state-commit] sc-subspace-max-pairs` and `sc-subspace-max-bytes` before upgrading. Values `<= 0` resolve to these defaults; there is no unlimited setting.
 * [#3927](https://github.com/sei-protocol/sei-chain/pull/3927) **Legacy Sei JSON-RPC and CLI removal.** Removes `sei_associate`, `sei_getBlockByHash`, `sei_getBlockByHashExcludeTraceFail`, `sei_getBlockTransactionCountByHash`, `sei_getBlockTransactionCountByNumber`, `sei_getEvmTx`, `sei_getFilterChanges`, `sei_getFilterLogs`, `sei_getLogs`, `sei_getTransactionByBlockHashAndIndex`, `sei_getTransactionByBlockNumberAndIndex`, `sei_getTransactionByHash`, `sei_getTransactionCount`, `sei_getTransactionErrorByHash`, `sei_getTransactionReceiptExcludeTraceFail`, `sei_getVMError`, `sei_newBlockFilter`, `sei_newFilter`, `sei_sign`, and `sei_uninstallFilter`. Use standard `eth_*` methods for EVM-originated data and `seid tx evm native-associate <custom-message> -y` for address association. There is no block- or filter-level replacement for discovering Cosmos-originated synthetic logs; clients that know the synthetic transaction hash can enable `sei_getTransactionReceipt`.
 

config/cosmosbase/agreement_test.go

@@ -109,6 +109,10 @@ func readerValues(t *testing.T) map[string]string {
 		"grpc.keepalive-timeout":               fmt.Sprint(cfg.GRPC.KeepaliveTimeout),
 		"grpc.keepalive-min-time":              fmt.Sprint(cfg.GRPC.KeepaliveMinTime),
 		"grpc.keepalive-permit-without-stream": fmt.Sprint(cfg.GRPC.KeepalivePermitWithoutStream),
+		"grpc.ip-rate-limit-rps":               fmt.Sprint(cfg.GRPC.IPRateLimitRPS),
+		"grpc.ip-rate-limit-burst":             fmt.Sprint(cfg.GRPC.IPRateLimitBurst),
+		"grpc.rate-limiting-enabled":           fmt.Sprint(cfg.GRPC.RateLimitingEnabled),
+		"grpc.trusted-proxy-cidrs":             fmt.Sprint(cfg.GRPC.TrustedProxyCIDRs),
 		"telemetry.service-name":               fmt.Sprint(cfg.Telemetry.ServiceName),
 		"telemetry.enabled":                    fmt.Sprint(cfg.Telemetry.Enabled),
 		"telemetry.enable-hostname":            fmt.Sprint(cfg.Telemetry.EnableHostname),

config/cosmosbase/cosmosbase.go

@@ -96,7 +96,7 @@ func apiDefaults(mode registry.Mode) any { return forMode(mode).API }
 // interface follows and for the same reason. The upstream default is on for every kind, so declaring that
 // would state an open interface on the nodes meant to expose the least.
 //
-// Six of these eleven keys are read only when the key is present. Two more are durations read through a
+// Nine of these fifteen keys are read only when the key is present. Two more are durations read through a
 // clamp that rescues a negative value and does nothing for an absent one, so those two are unguarded and
 // their clobber leaves no trace. The durations are declared as durations and written into a file as text,
 // which is the shape the reader parses back.

config/cosmosbase/cosmosbase_test.go

@@ -77,6 +77,8 @@ func TestTheGRPCKeysAreTheOnesItsReaderResolves(t *testing.T) {
 		"grpc.max-connection-idle", "grpc.max-connection-age", "grpc.max-connection-age-grace",
 		"grpc.keepalive-time", "grpc.keepalive-timeout", "grpc.keepalive-min-time",
 		"grpc.keepalive-permit-without-stream",
+		"grpc.ip-rate-limit-rps", "grpc.ip-rate-limit-burst", "grpc.rate-limiting-enabled",
+		"grpc.trusted-proxy-cidrs",
 	})
 }
 

ratelimiter/method_bucket.go

@@ -5,6 +5,8 @@ import "strings"
 const (
 	// PlaneCometBFT is the rate-limit plane label for Tendermint RPC HTTP.
 	PlaneCometBFT = "cometbft"
+	// PlaneGRPC is the rate-limit plane label for native gRPC (:9090).
+	PlaneGRPC = "grpc"
 
 	// rpcMethodBucketOther is the fallback label for unrecognized methods.
 	rpcMethodBucketOther = "other"
@@ -77,17 +79,32 @@ var knownCometBFTRPCMethods = map[string]struct{}{
 
 // bucketRPCMethod maps a raw JSON-RPC method name to a low-cardinality label
 // suitable for OTel/Prometheus metrics. Attacker-controlled method strings
-// collapse to rpcMethodBucketOther.
-func bucketRPCMethod(plane, method string) string {
+// collapse to rpcMethodBucketOther. knownGRPCMethods bounds the PlaneGRPC label
+// set; the other planes ignore it.
+func bucketRPCMethod(plane, method string, knownGRPCMethods map[string]struct{}) string {
 	if method == MethodInvalid {
 		return MethodInvalid
 	}
 	if plane == PlaneCometBFT {
 		return bucketCometBFTRPCMethod(method)
 	}
+	if plane == PlaneGRPC {
+		return bucketGRPCMethod(method, knownGRPCMethods)
+	}
 	return bucketNamespacedRPCMethod(method)
 }
 
+func bucketGRPCMethod(fullMethod string, known map[string]struct{}) string {
+	if fullMethod == "" || len(fullMethod) > maxRPCMethodLen {
+		return rpcMethodBucketOther
+	}
+	name := strings.TrimPrefix(fullMethod, "/")
+	if _, ok := known[name]; ok {
+		return name
+	}
+	return rpcMethodBucketOther
+}
+
 func bucketCometBFTRPCMethod(method string) string {
 	if method == "" || len(method) > maxRPCMethodLen {
 		return rpcMethodBucketOther

ratelimiter/method_bucket_test.go

@@ -8,21 +8,21 @@ import (
 )
 
 func TestBucketRPCMethod_KnownNamespaces(t *testing.T) {
-	require.Equal(t, "eth", bucketRPCMethod("evm", "eth_call"))
-	require.Equal(t, "eth", bucketRPCMethod("evm", "eth_getBalance"))
-	require.Equal(t, "debug", bucketRPCMethod("evm", "debug_traceTransaction"))
-	require.Equal(t, "web3", bucketRPCMethod("evm", "web3_clientVersion"))
+	require.Equal(t, "eth", bucketRPCMethod("evm", "eth_call", nil))
+	require.Equal(t, "eth", bucketRPCMethod("evm", "eth_getBalance", nil))
+	require.Equal(t, "debug", bucketRPCMethod("evm", "debug_traceTransaction", nil))
+	require.Equal(t, "web3", bucketRPCMethod("evm", "web3_clientVersion", nil))
 }
 
 func TestBucketRPCMethod_UnknownOrMalformed(t *testing.T) {
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", ""))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "notnamespaced"))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "eth"))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "_eth_call"))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "eth_"))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "bogus_method"))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "eth_BAD-chars"))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", strings.Repeat("a", maxRPCMethodLen+1)))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "notnamespaced", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "eth", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "_eth_call", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "eth_", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "bogus_method", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", "eth_BAD-chars", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod("evm", strings.Repeat("a", maxRPCMethodLen+1), nil))
 }
 
 func TestBucketRPCMethod_ValidAndInvalidMethods(t *testing.T) {
@@ -32,7 +32,7 @@ func TestBucketRPCMethod_ValidAndInvalidMethods(t *testing.T) {
 		"eth_random-uuid-1",
 		"eth_random-uuid-2",
 	} {
-		seen[bucketRPCMethod("evm", method)] = struct{}{}
+		seen[bucketRPCMethod("evm", method, nil)] = struct{}{}
 	}
 	require.Len(t, seen, 2)
 	require.Contains(t, seen, "eth")
@@ -48,17 +48,37 @@ func TestBucketRPCMethod_CometBFTKnownMethods(t *testing.T) {
 		"catalog",
 		"websocket",
 	} {
-		require.Equal(t, method, bucketRPCMethod(PlaneCometBFT, method), method)
+		require.Equal(t, method, bucketRPCMethod(PlaneCometBFT, method, nil), method)
 	}
 }
 
 func TestBucketRPCMethod_CometBFTUnknownMethods(t *testing.T) {
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneCometBFT, ""))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneCometBFT, "bogus"))
-	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneCometBFT, strings.Repeat("a", maxRPCMethodLen+1)))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneCometBFT, "", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneCometBFT, "bogus", nil))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneCometBFT, strings.Repeat("a", maxRPCMethodLen+1), nil))
 }
 
 func TestBucketRPCMethod_Invalid(t *testing.T) {
-	require.Equal(t, MethodInvalid, bucketRPCMethod("evm", MethodInvalid))
-	require.Equal(t, MethodInvalid, bucketRPCMethod(PlaneCometBFT, MethodInvalid))
+	require.Equal(t, MethodInvalid, bucketRPCMethod("evm", MethodInvalid, nil))
+	require.Equal(t, MethodInvalid, bucketRPCMethod(PlaneCometBFT, MethodInvalid, nil))
+	require.Equal(t, MethodInvalid, bucketRPCMethod(PlaneGRPC, MethodInvalid, nil))
+}
+
+func TestBucketRPCMethod_GrpcKnownMethods(t *testing.T) {
+	known := map[string]struct{}{
+		"cosmos.bank.v1beta1.Query/Balance":  {},
+		"cosmos.tx.v1beta1.Service/Simulate": {},
+	}
+	require.Equal(t, "cosmos.bank.v1beta1.Query/Balance", bucketRPCMethod(PlaneGRPC, "/cosmos.bank.v1beta1.Query/Balance", known))
+	require.Equal(t, "cosmos.tx.v1beta1.Service/Simulate", bucketRPCMethod(PlaneGRPC, "/cosmos.tx.v1beta1.Service/Simulate", known))
+}
+
+func TestBucketRPCMethod_GrpcUnknownMethods(t *testing.T) {
+	known := map[string]struct{}{"cosmos.bank.v1beta1.Query/Balance": {}}
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneGRPC, "", known))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneGRPC, "/cosmos.bank.v1beta1.Query/AllBalances", known))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneGRPC, "/bogus.Service/Call", known))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneGRPC, "not-a-grpc-path", known))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneGRPC, strings.Repeat("a", maxRPCMethodLen+1), known))
+	require.Equal(t, rpcMethodBucketOther, bucketRPCMethod(PlaneGRPC, "/cosmos.bank.v1beta1.Query/Balance", nil))
 }

ratelimiter/registry.go

@@ -7,6 +7,7 @@ import (
 	"net/http"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	"github.com/hashicorp/golang-lru/v2/expirable"
@@ -72,6 +73,25 @@ type Registry struct {
 	trustedProxies []*net.IPNet
 	lru            *expirable.LRU[string, *rate.Limiter]
 	mu             sync.Mutex
+	grpcMethods    atomic.Pointer[map[string]struct{}]
+}
+
+// SetKnownGRPCMethods bounds the method label recorded for PlaneGRPC rejections
+// to the given "service/Method" names. Call it before serving; unlisted methods
+// are recorded as "other".
+func (r *Registry) SetKnownGRPCMethods(methods []string) {
+	known := make(map[string]struct{}, len(methods))
+	for _, m := range methods {
+		known[strings.TrimPrefix(m, "/")] = struct{}{}
+	}
+	r.grpcMethods.Store(&known)
+}
+
+func (r *Registry) knownGRPCMethods() map[string]struct{} {
+	if known := r.grpcMethods.Load(); known != nil {
+		return *known
+	}
+	return nil
 }
 
 // New creates a Registry from cfg. Returns an error if any CIDR in TrustedProxyCIDRs is invalid.
@@ -101,7 +121,7 @@ func (r *Registry) Allow(ctx context.Context, ip, plane, method string) bool {
 		1,
 		metric.WithAttributes(
 			attribute.String("plane", plane),
-			attribute.String("method_namespace", bucketRPCMethod(plane, method)),
+			attribute.String("method_namespace", bucketRPCMethod(plane, method, r.knownGRPCMethods())),
 		),
 	)
 	return false
@@ -121,7 +141,7 @@ func (r *Registry) AllowN(ctx context.Context, ip, plane, method string, n int)
 		1,
 		metric.WithAttributes(
 			attribute.String("plane", plane),
-			attribute.String("method_namespace", bucketRPCMethod(plane, method)),
+			attribute.String("method_namespace", bucketRPCMethod(plane, method, r.knownGRPCMethods())),
 		),
 	)
 	return false
@@ -196,6 +216,9 @@ func (r *Registry) getOrCreate(ip string) *rate.Limiter {
 }
 
 // bucketKey returns the LRU key for ip.
+// The key is the address alone, so every client sharing an address shares one
+// bucket: all processes reaching the node over 127.0.0.1, and any population
+// behind a single egress address that TrustedProxyCIDRs does not cover.
 // IPv6 addresses are masked to /64: a client rotating within a /64 prefix
 // would otherwise get a fresh bucket per address.
 func bucketKey(ip string) string {

ratelimiter/registry_test.go

@@ -368,3 +368,39 @@ func TestNew_InvalidCIDR_ReturnsError(t *testing.T) {
 	})
 	require.ErrorContains(t, err, "bad")
 }
+
+func TestAllow_GRPCRejectionRecordsFullMethodMetric(t *testing.T) {
+	reader := sdkmetric.NewManualReader()
+	provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))
+	prev := otel.GetMeterProvider()
+	otel.SetMeterProvider(provider)
+	t.Cleanup(func() { otel.SetMeterProvider(prev) })
+
+	registryMetrics.rejectedCounter = must(provider.Meter("ratelimiter").Int64Counter(
+		"rpc_rate_limit_rejected_total",
+	))
+
+	r := mustNew(t, cfg(0.001, 1))
+	r.SetKnownGRPCMethods([]string{"/cosmos.bank.v1beta1.Query/Balance"})
+	ip := "10.0.0.42"
+	require.True(t, r.Allow(t.Context(), ip, PlaneGRPC, "/cosmos.bank.v1beta1.Query/Balance"))
+	require.False(t, r.Allow(t.Context(), ip, PlaneGRPC, "/cosmos.bank.v1beta1.Query/Balance"))
+
+	var rm metricdata.ResourceMetrics
+	require.NoError(t, reader.Collect(t.Context(), &rm))
+	found := false
+	for _, sm := range rm.ScopeMetrics {
+		for _, m := range sm.Metrics {
+			if m.Name != "rpc_rate_limit_rejected_total" {
+				continue
+			}
+			sum := m.Data.(metricdata.Sum[int64])
+			require.Equal(t, int64(1), sum.DataPoints[0].Value)
+			attrs := sum.DataPoints[0].Attributes.ToSlice()
+			require.Contains(t, attrs, attribute.String("plane", PlaneGRPC))
+			require.Contains(t, attrs, attribute.String("method_namespace", "cosmos.bank.v1beta1.Query/Balance"))
+			found = true
+		}
+	}
+	require.True(t, found, "expected rpc_rate_limit_rejected_total metric")
+}