[seanghaeli]

Since #71192, a worker swaps a deadline callback's single-use token for an execution token before running it (PATCH /execution/callbacks/{id}/run). If that request took longer than the 5 s client timeout, the client retried, the retry got a 409 because the token was already used, and the callback was marked failed without running. The same happened if the API was unreachable for more than about 15 s, for example during a restart.

This changes how that one call retries: it waits up to 60 s for an answer instead of retrying, and only retries when the request never reached the server (connection errors), for at least 2 minutes. The token stays single-use.

It still fails closed if the server takes over 60 s, or if the connection drops after the request was sent, since the server may already have used the token. That case now logs that the token exchange outcome is unknown, so it can be told apart from a callback that ran and failed.

# Breeze, Postgres, LocalExecutor, a SyncCallback that reads a Variable. A: the first exchange response is held 6 s
# after the server handled it. B: API server stopped from before the run until 30 s after it. C: connection dropped
# after the server handled the exchange.

# before (main)
[main3 21:34:45] A1 slow exchange (6s): callback failed output='Execution failed'; user code ran: 0 (cumulative)
[main3 21:35:43] A2 slow exchange (6s): callback failed output='Execution failed'; user code ran: 0 (cumulative)
[main3 21:37:40] B1 API down 30s at callback start: callback failed output='Execution failed'; user code ran: 0 (cumulative)
[main3 21:39:37] B2 API down 30s at callback start: callback failed output='Execution failed'; user code ran: 0 (cumulative)
[main3 21:40:49] C1 response lost after the exchange: callback failed output='Execution failed'; user code ran: 0 (cumulative)
Server error detail: {'reason': 'invalid_state', 'message': 'Callback 01a1ada1-f894-7ba4-941d-fd266a3efc60 is in state running; its token can only be exchanged once while QUEUED.', 'previous_state': 'running'}

# after
[fix3 21:25:04] A1 slow exchange (6s): callback success output=''; user code ran: 1 (cumulative)
[fix3 21:26:02] A2 slow exchange (6s): callback success output=''; user code ran: 2 (cumulative)
[fix3 21:27:59] B1 API down 30s at callback start: callback success output=''; user code ran: 3 (cumulative)
[fix3 21:29:56] B2 API down 30s at callback start: callback success output=''; user code ran: 4 (cumulative)
[fix3 21:31:09] C1 response lost after the exchange: callback failed output='Execution failed'; user code ran: 4 (cumulative)
httpx2.RemoteProtocolError: peer closed connection without sending complete message body (incomplete chunked read)

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5.5) following the guidelines

[amoghrajesh]

Thanks for catching this. Seems to be a real regression from #71192: but I have never seen such a delay from API server, though it could be possible. Lets get it in for 3.4.1

[amoghrajesh]

I think the fix should belong on the server side....

[amoghrajesh]

Isn't the root cause is on the server: PATCH /callbacks/{id}/run uses up the token on the first request, so no client retry is ever safe. If the endpoint returned the same execution token for a retry of the same callback within a short window, we would not need to tune timeouts. That would also cover 502/503 from a proxy / load balancer setup during restarts. Can we consider such an approach instead?

[seanghaeli]

Having the server hand out a token again would mean that anyone holding a copy of the callback token (it sits in the broker message) gets a new working execution token. That would defeat the purpose of #71192, and the callback token would no longer strictly be single-use.

With this client fix, the callback token remains strictly single-use, all that changes is that the client is careful about when it retries. The client knows which requests never made it to the server, and only retries those. The server only sees the ones that made it, so it can't tell a retry from someone replaying a copy.

For the lost response and the 502/503 during restarts, the client can't know if the token was used, so it fails the callback rather than retry. I can provide a follow-up that dispatches the callback again with a fresh token in these situations, but it would have a bigger review surface and might not make sense to backport to 3.4.1, which is why I'll treat it as a follow-up.