fix(coding-agent): defer and report heartbeat fires a busy session declines - #890
Open
Hotragn wants to merge 1 commit into
Open
fix(coding-agent): defer and report heartbeat fires a busy session declines#890Hotragn wants to merge 1 commit into
Hotragn wants to merge 1 commit into
Conversation
…clines A heartbeat that came due while its session was busy was consumed and lost. claimDueInState advances nextRunAt at claim time, before the busy check, and the skip branch of recordDispatchResult advanced it a second time from the skip time. Nothing re-armed the missed beat, an interval schedule lost its phase on every skip, and a busy window that spanned several beats left no trace beyond a single overwritten lastSkippedAt. Re-arm from the beat that was skipped instead of from the skip time, stepping the schedule forward until the next fire is strictly after now. That keeps the cadence, cannot make the scheduler spin on an already-due job, and yields the true number of fires the busy window swallowed. Those land in a new missedRunCount that accumulates across skips and resets on delivery. The delivered prompt was byte-identical on every beat: runCount and lastRunAt were computed into HeartbeatPromptDetails, which convertToLlm drops, so the model could not tell "beat 3 of 3" from "beat 3 of 40" and never learned that fires had been skipped. Heartbeat prompts now carry a <heartbeat> block with the beat number, schedule, previous delivery, and any backlog, ahead of the unchanged instruction. Queue previews still show the instruction alone. The backlog is also surfaced to humans: /heartbeat status renders "Missed: n since last run" and the daemon job listing renders missed=n. Publishing missedRunCount on cron/heartbeat job rows is an additive wire change, so DAEMON_SCHEMA_REVISION moves to 14; old clients ignore the field and new clients render nothing when an old daemon omits it. fixes PrimeIntellect-ai#820
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #820.
A heartbeat fire that lands on a busy session is currently consumed and lost, and the beats that do land carry no way to notice the gap.
The fire is consumed before the busy check
claimDueInStateadvancesjob.nextRunAtat claim time, beforerunCronJobever asksshouldDeferHeartbeatCronJob. When the session turns out to be busy,recordDispatchResult's skip branch advancesnextRunAta second time, fromnow. Three consequences:every 5mskipped 90s late becomesevery 5mmeasured from the skip, so the cadence drifts by the length of every busy check.lastSkippedAtis a single overwritten timestamp.rescheduleAfterSkipnow advances from the dispatch record'sscheduledFor— the beat that was actually skipped — and steps the schedule forward until the next fire is strictly afternow. So:every 5mdue at 12:05 and declined at 12:06:30 re-arms at 12:10, not 12:11:30.now, a busy session cannot make the scheduler spin on an already-due job. Dispatch attempts stay at one per interval, exactly as today.MAX_COALESCED_SKIPS); a pathologically short schedule against a very long busy window falls back to the plain advance while keeping the count it accumulated.Those fires land in a new
AgentCronJob.missedRunCount, which accumulates across skips and is cleared when a beat is delivered. All three skip paths feed it: therecordDispatchResultskip branch,recordSkipResult, andclaimDueInState's already-claimed branch (a fire arriving while the previous dispatch is still outstanding is swallowed the same way).I deliberately did not make a declined beat re-fire immediately. Re-arming at the original
scheduledForwould leave the job due while the session is still busy, andrunCronJobdeclines cheaply, so it would busy-loop for the length of the busy window. Coalescing forward preserves both the user's cadence and the scheduler's load profile.The delivered prompt could not report any of this
createHeartbeatPromptMessageputrunCount/lastRunAt/nextRunAtinHeartbeatPromptDetails, whichconvertToLlmdrops, and setcontent: job.prompt. Every delivery was byte-identical, so a heartbeat-driven agent could not tell "beat 3 of 3" from "beat 3 of 40".The heartbeat message content is what
convertToLlmturns into the provider's user message, soformatHeartbeatPromptContentnow prefixes the instruction with a compact block:The instruction stays last so it remains the final thing the model reads, and the skipped-fires line appears only when there is a backlog. Queue previews and
clearQueue()are unaffected — they read the prompt text passed to_promptInjectedMessage, which is still the bare instruction, soHeartbeat prompt: <instruction>renders exactly as before.Human-visible surfaces
/heartbeat statusgainedMissed: <n> since last run, most recent <date>; previously it showed nothing about skips at all.formatAgentCronJobgainedmissed=<n>alongside the existingskipped=<date>.Daemon protocol
DaemonCronJobis an alias ofAgentCronJob, somissedRunCountis published on cron/heartbeat job rows. That is an additive change on an existing response shape — a new daemon's extra field is ignored by an old client, and a new client renders nothing when an old daemon omits it — matching revisions 9/10/12, which also published new fields.DAEMON_SCHEMA_REVISIONmoves to 14 with a matching comment andDAEMON_SCHEMA_ID. The schema digest hashes thedaemon-protocol.tswire-type source only, so it is unchanged;test/daemon-protocol.test.tsverifies the identity stays synchronized.Tests
New:
packages/coding-agent/test/suite/regressions/820-heartbeat-missed-beats.test.ts(6 cases) — phase preservation on a declined beat, coalescing five fires across a 22-minute busy window with the re-armed fire proven to be in the future, accumulation across two skips followed by a delivery that clears the backlog,missed=in the daemon listing, the composed prompt text in first-delivery/backlog/singular forms, and an end-to-end case asserting the model actually receives the backlog notice.Updated (intentional behavior changes, not relaxations):
test/cron-jobs.test.ts— two tests asserted the drifting reschedule. One is renamed from "reschedules a skipped dispatch from the skip time" to "...on the original cadence"; both now also assertmissedRunCount.test/suite/regressions/4482-heartbeat-injected-prompt.test.ts— four assertions compared the delivered text to the bare instruction; they now compare againstformatHeartbeatPromptContent. The queue-preview andclearQueueassertions were left untouched and still pass, which is the check that previews stayed clean.Verified:
npm run checkclean;cron-jobs,daemon-protocol,daemon-client,agent-connection-daemon,messages,4482, and the new suite pass (180 + 6).daemon-mode.test.tshas 5 failures on this Windows machine (unix-socket bind and symlink cases) that reproduce identically on a cleanmainworktree.Note
Fix heartbeat scheduler to preserve cadence and track skipped fires on busy session declines
missedRunCounttally (capped atMAX_COALESCED_SKIPSintervals) that resets on the next successful delivery.<heartbeat>block with beat number, schedule, previous delivery time, and skipped-fire backlog when applicable./heartbeatstatus UI, daemon job listing (missed=<n>), and heartbeat message details (missedRunCount,lastSkippedAt).recordSkipResultandrecordDispatchResultnow acceptscheduledForand userescheduleAfterSkipinstead of advancing from the current time, changingnextRunAtbehavior for skipped jobs.Macroscope summarized 9df47d3.