Skip to content

Relay never starts on a caught-up target, and retries fetch errors with no backoff #48

Open

Description

@omkar619-dev

Hi! While testing on main (2dcedd3) I ran into two problems that seem to come from the same comparison in setWeight, so I'm filing them together.

1. A relay never starts while the target is caught up. If the target already holds as many messages as the source (both sides had 9 in my test), the relay logs no healthy server found. waiting and retrying (no healthy node) and never creates a consumer. With --stop-at-end it never exits: my run was killed after 20 seconds, with 4 of those lines and no "reached end of offsets". Without --stop-at-end it waits until a new message arrives on the source.

2. Fetch errors are retried without any backoff. When every fetch from the current source fails (I got there with OFFSET_OUT_OF_RANGE, as in zerodha/kaf-relay#43), the relay reconnects as fast as it can: 3,380 times in 23 seconds on the branch of zerodha/kaf-relay#47, with backoff_min = "2s". On v2.1.0 it's the same loop, and each attempt also leaves the previous client open, which that PR already fixes.

Why it seems to happen: at startup, setInitialOffsets sets a placeholder candidate that is marked unhealthy and weighs as much as the target (source_pool.go L182-L185), and setWeight only replaces the current candidate when the new weight is higher (L602).

  • A source with the same total as the target never replaces the placeholder, so getCurCandidate keeps returning ErrorNoHealthy (L557-L559), even though its own resume check accepts a source that has caught up (the < at L574).
  • GetFetches marks a failing server with setWeight(id, -1) (L264), but -1 is never higher, so the current candidate stays healthy, Get hands it straight back, and the backoff in Get (L241) never runs.

You know the failover rules much better than I do, so I'm only describing what I saw. Happy to test a fix if that helps.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions