Skip to content

fix(coding-agent): defer and report heartbeat fires a busy session declines - #890

Open
Hotragn wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
Hotragn:fix/820-heartbeat-missed-beats
Open

fix(coding-agent): defer and report heartbeat fires a busy session declines#890
Hotragn wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
Hotragn:fix/820-heartbeat-missed-beats

Conversation

@Hotragn

@Hotragn Hotragn commented Aug 7, 2026

Copy link
Copy Markdown

Fixes #820.

A heartbeat fire that lands on a busy session is currently consumed and lost, and the beats that do land carry no way to notice the gap.

The fire is consumed before the busy check

claimDueInState advances job.nextRunAt at claim time, before runCronJob ever asks shouldDeferHeartbeatCronJob. When the session turns out to be busy, recordDispatchResult's skip branch advances nextRunAt a second time, from now. Three consequences:

  • The missed beat is never re-armed.
  • An interval schedule loses its phase on every skip: every 5m skipped 90s late becomes every 5m measured from the skip, so the cadence drifts by the length of every busy check.
  • A busy window spanning several beats is indistinguishable from one skipped beat. lastSkippedAt is a single overwritten timestamp.

rescheduleAfterSkip now advances from the dispatch record's scheduledFor — the beat that was actually skipped — and steps the schedule forward until the next fire is strictly after now. So:

  • every 5m due at 12:05 and declined at 12:06:30 re-arms at 12:10, not 12:11:30.
  • The same job declined at 12:27 re-arms at 12:30 and reports 5 swallowed fires (12:05, 12:10, 12:15, 12:20, 12:25).
  • Because the result is always strictly after now, a busy session cannot make the scheduler spin on an already-due job. Dispatch attempts stay at one per interval, exactly as today.
  • The step loop is bounded (MAX_COALESCED_SKIPS); a pathologically short schedule against a very long busy window falls back to the plain advance while keeping the count it accumulated.

Those fires land in a new AgentCronJob.missedRunCount, which accumulates across skips and is cleared when a beat is delivered. All three skip paths feed it: the recordDispatchResult skip branch, recordSkipResult, and claimDueInState's already-claimed branch (a fire arriving while the previous dispatch is still outstanding is swallowed the same way).

I deliberately did not make a declined beat re-fire immediately. Re-arming at the original scheduledFor would leave the job due while the session is still busy, and runCronJob declines cheaply, so it would busy-loop for the length of the busy window. Coalescing forward preserves both the user's cadence and the scheduler's load profile.

The delivered prompt could not report any of this

createHeartbeatPromptMessage put runCount/lastRunAt/nextRunAt in HeartbeatPromptDetails, which convertToLlm drops, and set content: job.prompt. Every delivery was byte-identical, so a heartbeat-driven agent could not tell "beat 3 of 3" from "beat 3 of 40".

The heartbeat message content is what convertToLlm turns into the provider's user message, so formatHeartbeatPromptContent now prefixes the instruction with a compact block:

<heartbeat>
beat 7
schedule every 5m
previous delivery 2026-01-01T12:05:00.000Z
2 scheduled fires skipped while this session was busy (most recent 2026-01-01T12:15:00.000Z)
</heartbeat>

Check the deployment and report meaningful changes

The instruction stays last so it remains the final thing the model reads, and the skipped-fires line appears only when there is a backlog. Queue previews and clearQueue() are unaffected — they read the prompt text passed to _promptInjectedMessage, which is still the bare instruction, so Heartbeat prompt: <instruction> renders exactly as before.

Human-visible surfaces

/heartbeat status gained Missed: <n> since last run, most recent <date>; previously it showed nothing about skips at all. formatAgentCronJob gained missed=<n> alongside the existing skipped=<date>.

Daemon protocol

DaemonCronJob is an alias of AgentCronJob, so missedRunCount is published on cron/heartbeat job rows. That is an additive change on an existing response shape — a new daemon's extra field is ignored by an old client, and a new client renders nothing when an old daemon omits it — matching revisions 9/10/12, which also published new fields. DAEMON_SCHEMA_REVISION moves to 14 with a matching comment and DAEMON_SCHEMA_ID. The schema digest hashes the daemon-protocol.ts wire-type source only, so it is unchanged; test/daemon-protocol.test.ts verifies the identity stays synchronized.

Tests

New: packages/coding-agent/test/suite/regressions/820-heartbeat-missed-beats.test.ts (6 cases) — phase preservation on a declined beat, coalescing five fires across a 22-minute busy window with the re-armed fire proven to be in the future, accumulation across two skips followed by a delivery that clears the backlog, missed= in the daemon listing, the composed prompt text in first-delivery/backlog/singular forms, and an end-to-end case asserting the model actually receives the backlog notice.

Updated (intentional behavior changes, not relaxations):

  • test/cron-jobs.test.ts — two tests asserted the drifting reschedule. One is renamed from "reschedules a skipped dispatch from the skip time" to "...on the original cadence"; both now also assert missedRunCount.
  • test/suite/regressions/4482-heartbeat-injected-prompt.test.ts — four assertions compared the delivered text to the bare instruction; they now compare against formatHeartbeatPromptContent. The queue-preview and clearQueue assertions were left untouched and still pass, which is the check that previews stayed clean.

Verified: npm run check clean; cron-jobs, daemon-protocol, daemon-client, agent-connection-daemon, messages, 4482, and the new suite pass (180 + 6). daemon-mode.test.ts has 5 failures on this Windows machine (unix-socket bind and symlink cases) that reproduce identically on a clean main worktree.

Note

Fix heartbeat scheduler to preserve cadence and track skipped fires on busy session declines

  • When a heartbeat fire is declined because a session is busy, the scheduler now re-arms from the original scheduled beat time rather than the skip time, preserving the original cadence phase.
  • Multiple fires declined while a session is busy are coalesced into a missedRunCount tally (capped at MAX_COALESCED_SKIPS intervals) that resets on the next successful delivery.
  • Delivered heartbeat prompts now include a structured <heartbeat> block with beat number, schedule, previous delivery time, and skipped-fire backlog when applicable.
  • The missed count is surfaced in the /heartbeat status UI, daemon job listing (missed=<n>), and heartbeat message details (missedRunCount, lastSkippedAt).
  • Behavioral Change: recordSkipResult and recordDispatchResult now accept scheduledFor and use rescheduleAfterSkip instead of advancing from the current time, changing nextRunAt behavior for skipped jobs.

Macroscope summarized 9df47d3.

…clines

A heartbeat that came due while its session was busy was consumed and lost.
claimDueInState advances nextRunAt at claim time, before the busy check, and
the skip branch of recordDispatchResult advanced it a second time from the
skip time. Nothing re-armed the missed beat, an interval schedule lost its
phase on every skip, and a busy window that spanned several beats left no
trace beyond a single overwritten lastSkippedAt.

Re-arm from the beat that was skipped instead of from the skip time, stepping
the schedule forward until the next fire is strictly after now. That keeps the
cadence, cannot make the scheduler spin on an already-due job, and yields the
true number of fires the busy window swallowed. Those land in a new
missedRunCount that accumulates across skips and resets on delivery.

The delivered prompt was byte-identical on every beat: runCount and lastRunAt
were computed into HeartbeatPromptDetails, which convertToLlm drops, so the
model could not tell "beat 3 of 3" from "beat 3 of 40" and never learned that
fires had been skipped. Heartbeat prompts now carry a <heartbeat> block with
the beat number, schedule, previous delivery, and any backlog, ahead of the
unchanged instruction. Queue previews still show the instruction alone.

The backlog is also surfaced to humans: /heartbeat status renders "Missed: n
since last run" and the daemon job listing renders missed=n.

Publishing missedRunCount on cron/heartbeat job rows is an additive wire
change, so DAEMON_SCHEMA_REVISION moves to 14; old clients ignore the field
and new clients render nothing when an old daemon omits it.

fixes PrimeIntellect-ai#820
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Heartbeat fires that coincide with a busy session are dropped, not deferred, and the delivered prompt has no run counter or skip notice

1 participant