[Bug]: A scheduled run that fails is recorded as succeeded #17889

Activity

1. juliusmarminge commented on Oct 10, 2026

   

   juliusmarminge

   on Oct 10, 2026

   Member

   More actions

   Note

   Grok responding on behalf of Julius.

   Thanks for the report. I checked this against main and it matches what the code does.

   Where it happens: runTask in apps/server/src/scheduledTasks/ScheduledTaskService.ts (around lines 795–860 on main). It calls threadLaunch.launch(...) for unbound tasks or threadManagement.sendToThread(..., mode: "queue") for bound ones, wrapped in Effect.exit. Right after that it sets lastRunStatus = result._tag === "Success" ? "succeeded" : "failed", and markCompleted writes that status along with lastRunAt, nextRunAt and runCount + 1.

   So "succeeded" means the command was accepted/dispatched, not that the turn reached a terminal outcome. A provider rejection such as a usage limit happens later, during the turn, so the task still records success. Nothing in this path retries or catches up a run either. The webhook path reads the same status ("The run failed to start."), so webhook delivery logs have the same blind spot.

   Possible fix (unverified):

   • Record a dispatched state (running/dispatched) at dispatch time. Then have the run's terminal outcome (turn completed / failed / interrupted, matched by the scheduled-task: commandId or messageId / scheduledTaskId) update lastRunStatus and lastRunError once it lands.
   • Or keep the current shape and add a follow-up subscription that downgrades to failed with the provider error when that turn fails.
   • Retrying or notifying on a provider-limit failure is a separate product decision. A usage limit won't clear on an immediate retry.

   Tests such as ThreadLaunchService.test.ts and ClaudeAutomaticDelivery.integration.test.ts currently assert succeeded right after dispatch, so they'd need updating.

   Possibly related: #16183, #17309, #16818, #15553, #15167, #17458. I haven't checked how much each one overlaps.
2. juliusmarminge

   added

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   on Oct 10, 2026
3. SotoAugusto commented on Oct 30, 2026

   

   SotoAugusto

   on Oct 30, 2026

   More actions

   Another case, with numbers, from T3 Code on Linux. It happened on 2026-10-05, before #18467.

   A thread-bound recurring task fired every 3 minutes. The thread had moved from Codex to Claude. Each wake started a Codex run, and each run failed in under 1 s with "Insufficient context allowance for the provider handoff".

   • 315 consecutive wakes failed, from 2026-10-05 12:37 UTC to 2026-10-06 05:02 UTC (about 16.5 h).
   • Nothing stopped the task. I found the loop later, in an audit of the run history (orchestration_v2_projection_runs, status failed).

   #18467 fixes the provider choice. The loop is still possible for any wake that fails at once: a usage limit, the handoff budget, or a missing base branch as in #16183.

   Request: after the real terminal status is recorded, pause a recurring task after N consecutive failed wakes (for example 3). Show a notification with the last error.
4. ilshat-galimov commented on Oct 30, 2026

   

   ilshat-galimov

   on Oct 30, 2026

   More actions

   Another path to the same result: an unbound task (fresh thread per run) whose run fails in workspace preparation (git worktree add failed (ref_not_found), see #16183). ThreadLaunchService.launch returns once the thread exists and preparation runs in the background (prepareInBackground, forked into preparationScope), so runTask records succeeded and advances next_run_at by a full period (weekly in my case) before preparation fails. The error is visible only inside the created thread.
5. psicilia5179 commented on Oct 31, 2026

   

   psicilia5179

   on Oct 31, 2026

   More actions

   Another case of the workspace-preparation path described above, this time on a stable release, with the run row and the task row side by side.

   T3 Code 0.1.5 stable, Linux, t3 serve, Claude agent. A nightly unbound task was created with schedule_task in a repo that has master and origin/master and no main. Its first run:

   • Run: requested 06:00:00.501, completed 06:00:01.533, status: "failed", startedAt: null, workspacePreparation: {"type":"worktree","baseRef":"main","startFromOrigin":true}
   • Error item on that run: Workspace preparation failed during provision worktree: Git command failed in GitVcsDriver.createWorktree.createBranch (<repo>): git worktree add failed (ref_not_found). The base-ref cause is [Bug]: Scheduled tasks that launch a fresh thread always branch from main and fail every run in repos without it #16183.
   • Task row after the run: last_run_status: "succeeded", last_run_error: null, run_count: 1, still enabled, next_run_at set to the next night

   The agent never started. So this "succeeded" is false even under the narrower meaning that run_scheduled_task_now documents ("Success means started, not finished").

   At v0.1.5, runTask records succeeded as soon as threadLaunch.launch returns (ScheduledTaskService.ts L885-L889). launch forks preparation with Effect.forkIn(preparationScope) and returns before preparation finishes (ThreadLaunchService.ts L795-L806). When preparation fails, prepared-run.fail emits an error turn item and run.updated with status: "failed" for the run (Orchestrator.ts L9621-L9715). Nothing reports back to the task. The code is the same at v0.1.6 and at main 15fe283, so neither fixes it.

   Two open PRs touch this. #19774 targets the preparation failure directly. It hasn't moved since 10-19 and now conflicts with main. #23131, which closes this issue, follows run events instead. I read its diff at 1c4a18d but didn't run it. As far as I can tell, TrackedRuns.observe would settle this case as failed: the error item supplies failure.message, and run.updated with failed ends the run, so lastRunError would carry the preparation error. None of its tests cover a failed worktree preparation, though. A test for that case would settle whether #23131 also covers what #19774 fixes.

   Workaround in use: bind the task (bindToCurrentThread: true, which stores {"type":"root"}), and check the run's thread instead of lastRunStatus.

   Investigated with Claude Opus 5.5 (Claude Code) via T3 Code triage.
6. royalaid commented on Nov 3, 2026

   

   royalaid

   on Nov 3, 2026

   More actions

   Another confirmation, on Linux and Windows hosts (2026-10-30 and 2026-11-01). A thread-bound one-shot task and a daily task each showed lastRunStatus: "succeeded"; the bound thread's run failed about 2 s later because the provider's weekly limit was reached. The one-shot task never disabled itself. Only dispatch-time errors, such as ThreadManagementProviderInstanceDisabledError, showed failed.

   On upstream main (61c3794) this is apps/server/src/scheduledTasks/ScheduledTaskService.ts around line 957: runSucceeded is result._tag === "Success" for threadLaunch.launch or sendToThread, so it measures dispatch, not the run's outcome.

   A related gap: a thread-bound task runs on its thread's providerInstanceId. When that instance is disabled, nothing warns until the task fires. Flagging such tasks in list_scheduled_tasks would help.

   Our workaround: read the bound thread's recent runs instead of lastRunStatus.
7. kamtS commented on Nov 5, 2026

   

   kamtS

   on Nov 5, 2026

   More actions

   Another path to the same bug, on stable v0.1.6 (macOS arm64, desktop app, Claude agent). This one is a workspace-preparation failure, not a provider rejection, on a task that opens a new thread per run.

   Setup: a fixed-time scheduled task (19:30 daily) with bindToCurrentThread: false, created through the schedule_task MCP tool. It's in a project whose root is a plain folder with git repos inside it, so the root itself isn't a git repo. The task got the usual {"type":"worktree","baseRef":"main","startFromOrigin":true} (#16183).

   What happened: the 2026-11-04 08:30 UTC run failed before any provider turn, with this error:

   Workspace preparation failed during provision worktree: Git command failed in GitWorkflowService.resolveWorktreeBaseRemote (<project root>): Failed to resolve the VCS driver for this Git command.
   

   The scheduled_tasks row afterwards:

   field	value
   last_run_at	2026-11-04T08:30:03.849Z
   last_run_status	succeeded
   last_run_error	(empty)
   run_count	1

   So it isn't just provider rejections: a run that fails at workspace preparation is recorded as succeeded too. Nobody noticed until the user opened the thread the next morning, which meant one of the twice-daily jobs had silently never run.

   Seen in v0.1.6 source (2359643c): ThreadLaunchService.checkWorktreeBase wraps its git probes in Effect.orElseSucceed(() => undefined). A project with no git therefore passes the pre-launch check and only fails later, in provision-worktree. That may be why this failure skips the status write-back.

   Workaround: register the actual repo as its own project and recreate the task there.

   Triaged with Claude Code (Claude Opus 5.5) in a T3 Code triage session.