You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Relay never starts on a caught-up target, and retries fetch errors with no backoff #48
Hi! While testing on main (2dcedd3) I ran into two problems that seem to come from the same comparison in setWeight, so I'm filing them together.
1. A relay never starts while the target is caught up. If the target already holds as many messages as the source (both sides had 9 in my test), the relay logs no healthy server found. waiting and retrying (no healthy node) and never creates a consumer. With --stop-at-end it never exits: my run was killed after 20 seconds, with 4 of those lines and no "reached end of offsets". Without --stop-at-end it waits until a new message arrives on the source.
2. Fetch errors are retried without any backoff. When every fetch from the current source fails (I got there with OFFSET_OUT_OF_RANGE, as in zerodha/kaf-relay#43), the relay reconnects as fast as it can: 3,380 times in 23 seconds on the branch of zerodha/kaf-relay#47, with backoff_min = "2s". On v2.1.0 it's the same loop, and each attempt also leaves the previous client open, which that PR already fixes.
Why it seems to happen: at startup, setInitialOffsets sets a placeholder candidate that is marked unhealthy and weighs as much as the target (source_pool.go L182-L185), and setWeight only replaces the current candidate when the new weight is higher (L602).
A source with the same total as the target never replaces the placeholder, so getCurCandidate keeps returning ErrorNoHealthy (L557-L559), even though its own resume check accepts a source that has caught up (the < at L574).
GetFetches marks a failing server with setWeight(id, -1) (L264), but -1 is never higher, so the current candidate stays healthy, Get hands it straight back, and the backoff in Get (L241) never runs.
You know the failover rules much better than I do, so I'm only describing what I saw. Happy to test a fix if that helps.
Activity