rpc: WebSocket overload protection#20446
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds overload protections for JSON-RPC over WebSocket by (1) making DB read-tx acquisition fail fast for WS method calls and (2) enforcing a maximum concurrent WebSocket connection limit (HTTP 503 pre-upgrade), exposed via --ws.max.connections for both erigon and rpcdaemon.
Changes:
- Tag WebSocket connection contexts with
kv.WithNonBlockingAcquireand thread that base context through RPC dispatch/handler goroutines viaServeCodecWithContext. - Add a WebSocket connection limiter to the node HTTP server and a reusable
NewWSConnectionLimiterwrapper for rpcdaemon. - Add CLI/config plumbing and tests covering the new fail-fast overload mapping and connection limiting behavior.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| rpc/websocket.go | Tags WS connection context with kv.WithNonBlockingAcquire and uses ServeCodecWithContext. |
| rpc/server.go | Introduces ServeCodecWithContext and routes ServeCodec through it. |
| rpc/client.go | Allows initializing server-side connection handling with a provided base context. |
| rpc/websocket_test.go | Adds test asserting WS handler context is tagged and ErrReadTxLimitExceeded maps to JSON-RPC -32005. |
| node/rpcstack.go | Adds WsConnectionLimit support for built-in WS serving + exports NewWSConnectionLimiter. |
| node/rpcstack_test.go | Adds tests for both built-in WS connection limit and NewWSConnectionLimiter. |
| cmd/utils/flags.go | Adds --ws.max.connections flag definition. |
| node/cli/default_flags.go | Registers --ws.max.connections in default flags. |
| node/cli/flags.go | Wires --ws.max.connections into embedded rpcdaemon config. |
| cmd/rpcdaemon/cli/httpcfg/http_cfg.go | Adds WsMaxConnections to rpcdaemon HTTP config struct. |
| cmd/rpcdaemon/cli/config.go | Wires the flag into rpcdaemon and wraps WS handler with NewWSConnectionLimiter. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // of concurrent open connections. WsConnectionLimit enforces that here. | ||
| ws := h.wsHandler.Load().(*rpcHandler) | ||
| if ws != nil && isWebsocket(r) { | ||
| if checkPath(r, h.wsConfig.prefix) { | ||
| if h.wsConfig.WsConnectionLimit > 0 { | ||
| if h.wsConnCount.Add(1) > h.wsConfig.WsConnectionLimit { | ||
| h.wsConnCount.Add(-1) | ||
| wsConnectionRejected.Inc() | ||
| rpc.WriteOverloadedResponse(w) | ||
| return | ||
| } | ||
| defer h.wsConnCount.Add(-1) | ||
| } |
There was a problem hiding this comment.
wsConnCount is documented as tracking live WebSocket connections, but it is only incremented/decremented when WsConnectionLimit > 0. This makes the counter misleading (it will stay 0 when the limit is disabled). Consider either (a) tracking wsConnCount unconditionally for all WS connections, or (b) renaming/commenting it to reflect that it only supports the limiter path.
| // of concurrent open connections. WsConnectionLimit enforces that here. | |
| ws := h.wsHandler.Load().(*rpcHandler) | |
| if ws != nil && isWebsocket(r) { | |
| if checkPath(r, h.wsConfig.prefix) { | |
| if h.wsConfig.WsConnectionLimit > 0 { | |
| if h.wsConnCount.Add(1) > h.wsConfig.WsConnectionLimit { | |
| h.wsConnCount.Add(-1) | |
| wsConnectionRejected.Inc() | |
| rpc.WriteOverloadedResponse(w) | |
| return | |
| } | |
| defer h.wsConnCount.Add(-1) | |
| } | |
| // of concurrent open connections. wsConnCount tracks live connections, | |
| // and WsConnectionLimit enforces an upper bound when enabled. | |
| ws := h.wsHandler.Load().(*rpcHandler) | |
| if ws != nil && isWebsocket(r) { | |
| if checkPath(r, h.wsConfig.prefix) { | |
| connCount := h.wsConnCount.Add(1) | |
| if h.wsConfig.WsConnectionLimit > 0 && connCount > h.wsConfig.WsConnectionLimit { | |
| h.wsConnCount.Add(-1) | |
| wsConnectionRejected.Inc() | |
| rpc.WriteOverloadedResponse(w) | |
| return | |
| } | |
| defer h.wsConnCount.Add(-1) |
yperbasis
left a comment
There was a problem hiding this comment.
Issues
- (Medium) httpServer.ServeHTTP limit code is unreachable in production
The PR adds connection limiting in two places:
- Inline in httpServer.ServeHTTP (using wsConnCount + wsConfig.WsConnectionLimit)
- Standalone wsConnectionLimiter via NewWSConnectionLimiter
The httpServer.enableWS path is only called from tests. In production, both the embedded node and standalone rpcdaemon go through startRegularRpcServer, which uses NewWSConnectionLimiter. So the inline logic in
httpServer.ServeHTTP + the wsConnCount field + the WsConnectionLimit field on wsConfig are dead code in production.
This isn't a bug — the limit works correctly via NewWSConnectionLimiter — but it's duplicated infrastructure. Consider either:
- Removing the httpServer.ServeHTTP inline path and having enableWS wrap the handler with NewWSConnectionLimiter instead (DRY)
- Or documenting that the httpServer path is kept for future use / test coverage
-
(Low) Copilot's inline comment is valid: wsConnCount is only incremented when WsConnectionLimit > 0, making it useless as a general observability metric. Tracking it unconditionally (and gating only the
rejection) would make ws_connection_count a useful gauge even without a configured limit. -
(Low) No log line on rejection: Connection rejections only increment the ws_connection_rejected_total metric — no log.Warn. This is consistent with rpcAdmissionHandler, so it's fine by convention, but a
sampled/rate-limited log would help operators diagnosing issues without a metrics stack.
Minor nits
- rpc/websocket.go:72-73 — The comment "Tag the connection context so BeginRo fails fast" is helpful. Consider also noting that r.Context() survives post-Upgrade because the handler goroutine blocks until the
WS connection closes. - node/rpcstack_test.go:460 — TestWsConnectionLimit uses require.Eventually with a 1ms polling interval. Under CI load this is fine since gorilla/websocket upgrade is synchronous, but a 5ms interval would be
gentler.
Summary
The PR is in good shape. The core logic (NonBlockingAcquire propagation, atomic connection limiting, error remapping) is correct and well-tested. The main question is whether the dual implementation of
connection limiting (inline in httpServer.ServeHTTP + standalone wsConnectionLimiter) should be consolidated to reduce surface area. No blocking issues.
|
@andrew remove code duplication and minor issue. |
Two defences against WebSocket overload: 1. Fail-fast on DB semaphore full: WebsocketHandler now tags the connection context with kv.WithNonBlockingAcquire so every method call uses TryAcquire instead of blocking indefinitely. When the read-tx semaphore is exhausted, BeginRo returns ErrReadTxLimitExceeded immediately; remapDBOverload converts it to JSON-RPC -32005. ServeCodecWithContext is added to thread the context through the dispatch/handler goroutines without storing it in the Client struct. 2. Connection limiter: WsConnectionLimit on wsConfig / httpServer rejects new WebSocket connections with HTTP 503 once the limit is exceeded. NewWSConnectionLimiter exports the same logic as a standalone http.Handler wrapper, used by startRegularRpcServer in rpcdaemon. Exposed as --ws.max.connections CLI flag for both erigon and rpcdaemon (0 = unlimited). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hain Remove the duplicate inline limit check from httpServer.ServeHTTP and move connection limiting into the handler chain via wsConnectionLimiter, consistent with how production (startRegularRpcServer) already applies NewWSConnectionLimiter. The wsConnCount field is replaced by wsLimiter which tracks live connections unconditionally when a limit is configured. Also clarify that r.Context() remains valid for the full WebSocket session and bump the TestWsConnectionLimit polling interval to 5ms. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
9614e44 to
93a5b1e
Compare
Two defences against WebSocket overload:
Fail-fast on DB semaphore full: WebsocketHandler now tags the connection context with kv.WithNonBlockingAcquire so every method call uses TryAcquire instead of blocking indefinitely. When the read-tx semaphore is exhausted, BeginRo returns ErrReadTxLimitExceeded immediately; remapDBOverload converts it to JSON-RPC -32005. ServeCodecWithContext is added to thread the context through the dispatch/handler goroutines without storing it in the Client struct.
Connection limiter: WsConnectionLimit on wsConfig / httpServer rejects new WebSocket connections with HTTP 503 once the limit is exceeded. NewWSConnectionLimiter exports the same logic as a standalone http.Handler wrapper, used by startRegularRpcServer in rpcdaemon. Exposed as --ws.max.connections CLI flag for both erigon and rpcdaemon (0 = unlimited).