Skip to content

[Bug]: Every provider wake and delegated result is dispatched by one fiber, behind parent locks, so wakes stall for hours #37413

Closed

Description

@navidnadali

Note

Written by Claude Opus 5.5 (Claude Code) on behalf of Navid

Area

apps/server

Steps to reproduce

On main at e367ef0da1, three kinds of continuation go into ProviderContinuationRequests, a single unbounded queue:

ProviderContinuationService drains that queue with one fiber, one dispatch at a time (requests.take → dispatchContinuation
→ Effect.forever). A wake to a delegated child first takes the parent's dispatch lock (mayStartSubagentFollowup →
withSubagentParentLock).

  1. A coordinator thread holds its lock for a while. Under load this is common: see A send or queued start to a long thread still reads 6–15 s of its history on main (follow-up to #29468) #36127, and [Bug]: A delegated child's checkpoint capture reads every open delivery of its parent while holding the parent's lock #37393 before fix(server): a delegated child's turn end re-offers only its own parent run's wake #37399 fixed it.
  2. A continuation for one of its children reaches the head of the queue and waits for that lock.
  3. Every other continuation on the server, for any thread, waits behind it.

After a restart, recovery re-offers every open completion delivery through the same queue, so wakes also wait behind that wave.

Expected behavior

A wake for one thread is not held up by another thread's lock, or by a backlog of re-deliveries. Order is kept per thread.

Actual behavior

Production server, read-only from its trace and state database:

  • One fiber. All 1,189 dispatchContinuation spans in the captured traces belong to one trace, and no two overlap.
  • Dispatch times. The p50 is 89 ms, but ten dispatches took 9–308 s. Only 2–6 s of each appears in child spans; the rest is a
    wait with no span.
  • A busy fiber. It was busy 300 of 300 seconds in every 5-minute bucket of a 20-minute window.
  • No wakes after a restart. For 2 h 40 min, zero background-task wakes were dispatched, while about 1,277 delegated-completion
    re-deliveries went through. Before the restart there were 22–54 such wakes per 10 minutes.
  • Delegated threads working unseen. 140 delegated threads ran on their own, with no T3 run, for a median of 172 minutes.
    During that time their MCP writes failed with parent_not_active (delegate_task fails with parent_not_active when a Claude wake turn runs before T3 opens its run #33272).

Impact

On a busy server an agent's wake can take minutes to hours to become a run. Until then the agent's T3 tool calls are refused, and
to its parent it looks idle or dead.

Version or commit

The code path is unchanged on main at e367ef0da1. Measured on a build of main from 11-14 with local performance patches. In
ProviderContinuationService those patches only narrow a read in the retry path; the single consumer and the parent lock are the
same as on main.

Environment

Linux, Node 26, about 185 Claude CLIs, coordinator threads with long-lived delegated threads.

Workaround

None in the app.

Related

We have a fix in our patched build. It does two things:

In an in-process model of the load above, run on a test clock:

  • the longest stretch with no wake dispatched went from 140–170 minutes to none;
  • a wake whose family's lock was free was dispatched in about 0.1 s.

I can send the first part as a PR (it builds on the change for #37412), or a maintainer can take it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions