[Proposal] Connection count & error metrics

[rkistner]

Background

The service already exposes sync API metrics through a Prometheus endpoint (internally via OpenTelemetry libs):

• powersync_concurrent_connections
• powersync_data_synced_bytes_total
• powersync_data_sent_bytes_total
• powersync_operations_synced_total

These metrics show current load and throughput, but they do not make it easy to answer:

• Are there recent error spikes?
• What percentage of sync connections end successfully?
• Which close reasons are increasing?
• Which PowerSync error codes are causing failed streams?
• Are failures specific to the HTTP streaming endpoint or the RSocket endpoint?

The sync routes already compute a best-effort close_reason for logs, so the main missing piece is a low-cardinality counter that records one final outcome per connection. Scanning the logs is a workaround sometimes used to get error rates, but it's not an efficient method for that, and requires custom work to generate alerts.

Proposal

Add a monotonic counter:

powersync_sync_connections_total

Description:

Total number of completed sync connections, labelled by outcome, close reason, error code, and transport.

Record exactly one increment when a sync stream finishes after it has been accepted as a sync connection. This should be paired with the existing powersync_concurrent_connections decrement sites so that each opened sync connection produces one final outcome.

Recommended labels:

Label	Values	Notes
outcome	success, error	Primary label for success/error rates.
close_reason	bounded route-defined enum	Reuse the existing close reason concept.
error_code	none, other, PSYNC_*	Use none for success, a PowerSync error code when available, and other for non-ServiceError failures.
transport	http_stream, rsocket	Separates /sync/stream HTTP from RSocket reactive streams.

Example Prometheus output:

powersync_sync_connections_total{outcome="success",close_reason="client_closed",error_code="none",transport="http_stream"} 1204
powersync_sync_connections_total{outcome="error",close_reason="stream_error",error_code="PSYNC_S2001",transport="http_stream"} 17
powersync_sync_connections_total{outcome="error",close_reason="service_unavailable",error_code="PSYNC_S2003",transport="rsocket"} 3

Close reason values

This should match what is produced in the logs, but we may need to reconsider the specific values for both.

Example values:

Value	Meaning
client_closed	The client closed the stream or request.
service_closed	The service ended the stream normally, for example auth expiry or switching sync config.
process_shutdown	The service is shutting down.
stream_error	The stream failed with an error while running.
service_unavailable	The request was rejected because the router is closed.
no_sync_config	The request was rejected because no sync config is active.
unknown	Fallback when no better reason is available.

Success and error classification

Classify outcome from the final stream state:

• success when the stream finishes without a thrown or reported error. This must specifically include cases like auth expiry, switching sync config, shutting down the process, or client disconnect.
• error when the stream returns a ServiceError, throws a non-service exception, or is rejected before streaming due to service state.

Suggested error_code handling:

• For ServiceError, use error.errorData.code. This includes InternalServerError.
• For non-service exceptions that are not wrapped, use other.
• For successful completions, use none.

Client disconnects should normally be outcome="success" with close_reason="client_closed" unless the server was already handling a stream error. This keeps normal mobile reconnect behavior from inflating error rates.

For error codes, we may need to revisit some existing error handling. For example, I've seen PSYNC_S2403 errors wrapped in a generic PSYNC_S2001 - we should remove that wrapping.

Also consider:

1. Splitting out PSYNC_S2403 into separate errors for timeout while loading data vs timeout while computing checksums.
2. Wrap at least some common MongoDB errors, such as connection timeout or "not primary".

Query examples

Success rate over five minutes:

sum(rate(powersync_sync_connections_total{outcome="success"}[5m]))
/
sum(rate(powersync_sync_connections_total[5m]))

Error rate by PowerSync error code:

sum by (error_code) (
  rate(powersync_sync_connections_total{outcome="error"}[5m])
)

Close reasons by transport:

sum by (transport, close_reason) (
  rate(powersync_sync_connections_total[5m])
)

Cardinality guidance

Do not add client_id, user_id, sync config id, bucket name, request parameters, or raw exception messages as labels. Those values are high-cardinality and belong in logs or traces instead.

The proposed label set should stay bounded:

• outcome: 2 values.
• transport: 2 values initially.
• close_reason: small enum controlled by code.
• error_code: bounded by the PowerSync error-code enum plus none and unknown.

Open questions

• Should rejected requests before powersync_concurrent_connections increments be counted by this metric? The recommendation is yes for sync-specific service-state rejections, such as PSYNC_S2003 and PSYNC_S2302, because they are connection attempts from the user's perspective. Authentication and validation failures may be better represented by a separate API request metric.
• Should close_reason="service_closed" be split further for auth expiry vs sync config switch? This would be useful if the route can identify those causes reliably without parsing generic stream completion.

AI Usage

This proposal was generated by Codex gpt-5.5, reviewed and edited manually.

[joshuabrink]

I ran this discussion through a couple rounds of Fable and gpt-5.5 against targeted past tickets and issues. I sifted through the suggestions and I think these points hold.

On the first open question. I'd go the other way and keep all pre-stream rejections out of this counter, including PSYNC_S2003/PSYNC_S2302. With service_unavailable and no_sync_config in the mix, success/(success+error) conflates streams that ran and failed with requests that never became a stream. The separate request metric you mentioned for auth/validation failures could cover all rejections instead, with a stage/reason breakdown:

powersync_sync_connections_total{transport,outcome,close_reason,error_code}
powersync_sync_connection_rejections_total{transport,stage,reason,error_code}

Pre-stream failures come up fairly often in support (PSYNC_S2305 before any data is sent, PSYNC_S2302 during restarts, auth failures before the stream is accepted), so I think a dedicated breakdown would be worthwhile.

[rkistner]

I'm happy in principle with that approach.

[joshuabrink]

I took another look at the connection lifecycle for the transports. The above did make sense, but the actual implementation does need a bit of work. For a minimal pass, I think we can get away with adding one additional value to outcome="rejected" and then just keep the rejection reasons in close_reason.

Accepted stream success rate would be outcome="success" / (outcome!="rejected"), total attempt failure rate is the unfiltered query, and rejections can't drag the accepted stream error rate around.

Here is the connection lifecycle outcome matrix, as I see it:

Condition	Lifecycle position	outcome	close_reason	error_code
Client disconnects accepted stream	termination	success	client_closed	none
Token expiry / sync-config switch	termination	success	service_closed	none
Rolling deploy / process shutdown	termination	success	process_shutdown	none
ServiceError mid-stream (e.g. S2305 too-many-buckets)	streaming	error	stream_error	PSYNC_*
Non-service exception mid-stream	streaming	error	stream_error	other
Router closed before accept	pre-increment	rejected	service_unavailable	PSYNC_S2003
No active sync config (null)	pre-increment	rejected	no_sync_config	PSYNC_S2302
(1) getActiveSyncConfig() storage query throws during setup	pre-increment, outside try	rejected	storage_error	PSYNC_S2402 / S2403 / raw
(2) Concurrency limit at acceptor	per-connection (rsocket only)	rejected	concurrency_limit	PSYNC_S2304

1. Emits no event at all today pre-increment and outside the try. Counting it would require moving the try up (or adding a catch).
2. Lives in the generic rsocket-router package, which has no metricsEngine, would need a hook/event rather than a one-line increment.

---

I think we defer auth and validation rejections, since it's more HTTP/request level concern, less sync-lifecycle concern + they fail before the code knows it's a sync attempt:

• HTTP: auth is shared middleware (authUser) that runs across all routes. Both fire before sync-stream.ts's handler body executes, so not yet "an accepted-or-rejected sync attempt."
• RSocket: auth is on the SETUP frame, once per WebSocket connection (in the acceptor), before any requestStream exists.

To count these as sync rejections we'd need route-aware hooks reaching into middleware/acceptor, which seems out of scope for a first pass. But I think this is where a separate counter would make more sense (something like: api_requests_total{route,status,error_code} ).

[rkistner]

These sound good to me for a start.

I think it would be good to eventually include the auth errors as well in the same metric as well - don't let our current code structure be the limiting factor here. But also happy if that's not part of the initial implementation.