Skip to content

rpc: WebSocket overload protection#20446

Merged
yperbasis merged 2 commits into
mainfrom
lupin012/ws_overload_protection
Apr 15, 2026
Merged

rpc: WebSocket overload protection#20446
yperbasis merged 2 commits into
mainfrom
lupin012/ws_overload_protection

Conversation

@lupin012

@lupin012 lupin012 commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Two defences against WebSocket overload:

  1. Fail-fast on DB semaphore full: WebsocketHandler now tags the connection context with kv.WithNonBlockingAcquire so every method call uses TryAcquire instead of blocking indefinitely. When the read-tx semaphore is exhausted, BeginRo returns ErrReadTxLimitExceeded immediately; remapDBOverload converts it to JSON-RPC -32005. ServeCodecWithContext is added to thread the context through the dispatch/handler goroutines without storing it in the Client struct.

  2. Connection limiter: WsConnectionLimit on wsConfig / httpServer rejects new WebSocket connections with HTTP 503 once the limit is exceeded. NewWSConnectionLimiter exports the same logic as a standalone http.Handler wrapper, used by startRegularRpcServer in rpcdaemon. Exposed as --ws.max.connections CLI flag for both erigon and rpcdaemon (0 = unlimited).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds overload protections for JSON-RPC over WebSocket by (1) making DB read-tx acquisition fail fast for WS method calls and (2) enforcing a maximum concurrent WebSocket connection limit (HTTP 503 pre-upgrade), exposed via --ws.max.connections for both erigon and rpcdaemon.

Changes:

  • Tag WebSocket connection contexts with kv.WithNonBlockingAcquire and thread that base context through RPC dispatch/handler goroutines via ServeCodecWithContext.
  • Add a WebSocket connection limiter to the node HTTP server and a reusable NewWSConnectionLimiter wrapper for rpcdaemon.
  • Add CLI/config plumbing and tests covering the new fail-fast overload mapping and connection limiting behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
rpc/websocket.go Tags WS connection context with kv.WithNonBlockingAcquire and uses ServeCodecWithContext.
rpc/server.go Introduces ServeCodecWithContext and routes ServeCodec through it.
rpc/client.go Allows initializing server-side connection handling with a provided base context.
rpc/websocket_test.go Adds test asserting WS handler context is tagged and ErrReadTxLimitExceeded maps to JSON-RPC -32005.
node/rpcstack.go Adds WsConnectionLimit support for built-in WS serving + exports NewWSConnectionLimiter.
node/rpcstack_test.go Adds tests for both built-in WS connection limit and NewWSConnectionLimiter.
cmd/utils/flags.go Adds --ws.max.connections flag definition.
node/cli/default_flags.go Registers --ws.max.connections in default flags.
node/cli/flags.go Wires --ws.max.connections into embedded rpcdaemon config.
cmd/rpcdaemon/cli/httpcfg/http_cfg.go Adds WsMaxConnections to rpcdaemon HTTP config struct.
cmd/rpcdaemon/cli/config.go Wires the flag into rpcdaemon and wraps WS handler with NewWSConnectionLimiter.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread node/rpcstack.go Outdated
Comment on lines +210 to +222
// of concurrent open connections. WsConnectionLimit enforces that here.
ws := h.wsHandler.Load().(*rpcHandler)
if ws != nil && isWebsocket(r) {
if checkPath(r, h.wsConfig.prefix) {
if h.wsConfig.WsConnectionLimit > 0 {
if h.wsConnCount.Add(1) > h.wsConfig.WsConnectionLimit {
h.wsConnCount.Add(-1)
wsConnectionRejected.Inc()
rpc.WriteOverloadedResponse(w)
return
}
defer h.wsConnCount.Add(-1)
}

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wsConnCount is documented as tracking live WebSocket connections, but it is only incremented/decremented when WsConnectionLimit > 0. This makes the counter misleading (it will stay 0 when the limit is disabled). Consider either (a) tracking wsConnCount unconditionally for all WS connections, or (b) renaming/commenting it to reflect that it only supports the limiter path.

Suggested change
// of concurrent open connections. WsConnectionLimit enforces that here.
ws := h.wsHandler.Load().(*rpcHandler)
if ws != nil && isWebsocket(r) {
if checkPath(r, h.wsConfig.prefix) {
if h.wsConfig.WsConnectionLimit > 0 {
if h.wsConnCount.Add(1) > h.wsConfig.WsConnectionLimit {
h.wsConnCount.Add(-1)
wsConnectionRejected.Inc()
rpc.WriteOverloadedResponse(w)
return
}
defer h.wsConnCount.Add(-1)
}
// of concurrent open connections. wsConnCount tracks live connections,
// and WsConnectionLimit enforces an upper bound when enabled.
ws := h.wsHandler.Load().(*rpcHandler)
if ws != nil && isWebsocket(r) {
if checkPath(r, h.wsConfig.prefix) {
connCount := h.wsConnCount.Add(1)
if h.wsConfig.WsConnectionLimit > 0 && connCount > h.wsConfig.WsConnectionLimit {
h.wsConnCount.Add(-1)
wsConnectionRejected.Inc()
rpc.WriteOverloadedResponse(w)
return
}
defer h.wsConnCount.Add(-1)

Copilot uses AI. Check for mistakes.

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issues

  1. (Medium) httpServer.ServeHTTP limit code is unreachable in production

The PR adds connection limiting in two places:

  • Inline in httpServer.ServeHTTP (using wsConnCount + wsConfig.WsConnectionLimit)
  • Standalone wsConnectionLimiter via NewWSConnectionLimiter

The httpServer.enableWS path is only called from tests. In production, both the embedded node and standalone rpcdaemon go through startRegularRpcServer, which uses NewWSConnectionLimiter. So the inline logic in
httpServer.ServeHTTP + the wsConnCount field + the WsConnectionLimit field on wsConfig are dead code in production.

This isn't a bug — the limit works correctly via NewWSConnectionLimiter — but it's duplicated infrastructure. Consider either:

  • Removing the httpServer.ServeHTTP inline path and having enableWS wrap the handler with NewWSConnectionLimiter instead (DRY)
  • Or documenting that the httpServer path is kept for future use / test coverage
  1. (Low) Copilot's inline comment is valid: wsConnCount is only incremented when WsConnectionLimit > 0, making it useless as a general observability metric. Tracking it unconditionally (and gating only the
    rejection) would make ws_connection_count a useful gauge even without a configured limit.

  2. (Low) No log line on rejection: Connection rejections only increment the ws_connection_rejected_total metric — no log.Warn. This is consistent with rpcAdmissionHandler, so it's fine by convention, but a
    sampled/rate-limited log would help operators diagnosing issues without a metrics stack.

Minor nits

  • rpc/websocket.go:72-73 — The comment "Tag the connection context so BeginRo fails fast" is helpful. Consider also noting that r.Context() survives post-Upgrade because the handler goroutine blocks until the
    WS connection closes.
  • node/rpcstack_test.go:460 — TestWsConnectionLimit uses require.Eventually with a 1ms polling interval. Under CI load this is fine since gorilla/websocket upgrade is synchronous, but a 5ms interval would be
    gentler.

Summary

The PR is in good shape. The core logic (NonBlockingAcquire propagation, atomic connection limiting, error remapping) is correct and well-tested. The main question is whether the dual implementation of
connection limiting (inline in httpServer.ServeHTTP + standalone wsConnectionLimiter) should be consolidated to reduce surface area. No blocking issues.

@lupin012

Copy link
Copy Markdown
Contributor Author

@andrew remove code duplication and minor issue.

lupin012 and others added 2 commits April 14, 2026 20:42
Two defences against WebSocket overload:

1. Fail-fast on DB semaphore full: WebsocketHandler now tags the
   connection context with kv.WithNonBlockingAcquire so every method
   call uses TryAcquire instead of blocking indefinitely. When the
   read-tx semaphore is exhausted, BeginRo returns ErrReadTxLimitExceeded
   immediately; remapDBOverload converts it to JSON-RPC -32005.
   ServeCodecWithContext is added to thread the context through the
   dispatch/handler goroutines without storing it in the Client struct.

2. Connection limiter: WsConnectionLimit on wsConfig / httpServer rejects
   new WebSocket connections with HTTP 503 once the limit is exceeded.
   NewWSConnectionLimiter exports the same logic as a standalone
   http.Handler wrapper, used by startRegularRpcServer in rpcdaemon.
   Exposed as --ws.max.connections CLI flag for both erigon and rpcdaemon
   (0 = unlimited).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…hain

Remove the duplicate inline limit check from httpServer.ServeHTTP and
move connection limiting into the handler chain via wsConnectionLimiter,
consistent with how production (startRegularRpcServer) already applies
NewWSConnectionLimiter. The wsConnCount field is replaced by wsLimiter
which tracks live connections unconditionally when a limit is configured.

Also clarify that r.Context() remains valid for the full WebSocket session
and bump the TestWsConnectionLimit polling interval to 5ms.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@lupin012 lupin012 force-pushed the lupin012/ws_overload_protection branch from 9614e44 to 93a5b1e Compare April 14, 2026 18:42
@yperbasis yperbasis added this pull request to the merge queue Apr 15, 2026
Merged via the queue into main with commit 8c4dd54 Apr 15, 2026
36 checks passed
@yperbasis yperbasis deleted the lupin012/ws_overload_protection branch April 15, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants