Repository navigation
Description
What happened
In a large workspace with many untracked files, every turn kept showing as working for about two minutes after the model finished. The delay is checkpoint capture. It grows with the total size of untracked files, not with what the turn changed.
Environment
- T3 Code (Beta) desktop, server bundle version 0.1.0, installed build as of 2026-11-15, Claude provider
- macOS 26.6.2, Apple M5, git 2.39.2
- One repository with about 42,000 tracked files, used mostly for documents. At the time it had 8,796 untracked files totalling 7.1 GB (Word files, images, PDFs).
- 7 or more threads running in place in the repository root, plus several threads in T3 worktrees.
Diagnosis
GitVcsDriver.checkpoints.captureCheckpoint creates a new private index for each capture (t3-checkpoint-index-<uuid> in the common dir). It seeds it by copying the user's index, runs read-tree --reset HEAD, then add -A -- ., and deletes the private index afterwards.
read-tree --reset HEAD keeps cached stat data only for entries that match HEAD. Untracked files are never in that index. So add -A has to read and hash the full content of every untracked file on every capture, even though nothing changed. Changed tracked files have the same problem.
A second cost stacks on top. orchestrationV2.checkpointCapture.execute takes the per-workspace lock up to three times per turn: twice in materializeBaselineCheckpoint and once in capture. With several threads in one workspace, each turn queues behind the other threads' 30-second captures. materializeBaselineCheckpoint only looks up a ref while it holds the lock.
Evidence
Server trace spans from server.trace.ndjson, real provider turns, same workspace and same session. "After" means after the untracked files were committed locally as a workaround, so they match HEAD and keep their stat data. Nothing else changed.
GitVcsDriver.checkpoints.captureCheckpoint- Before: n=61, p50 31.5 s, p90 33.2 s, max 38.7 s.
- After: n=72, p50 3.3 s, p90 15.2 s, max 32.0 s.
orchestrationV2.checkpointCapture.execute- Before: n=36, p50 116.2 s, p90 128.2 s, max 140.2 s.
- After: n=67, p50 4.0 s, p90 19.9 s, max 35.7 s.
RunFinalizationService.finalize- Before: n=36, p50 119.2 s, p90 130.1 s, max 144.3 s.
- After: n=66, p50 32.6 s (see below).
The remaining finalize time after the workaround comes from a separate status refresh, filed as #37437.
Replaying the capture by hand on the same repository, with the same -c core.fsmonitor=false -c core.fsync=objects,reference -c core.fsyncMethod=fsync add -A -- .:
- Fresh private index (copy, then
read-tree --reset HEAD, thenadd -A): 30.3 s - Running
add -Aa second time on that same private index: 0.9 s - Fresh private index after the untracked files were committed: 1.0 s
Prior work
#13543 proposed keeping the private index between captures and was closed with a request for an issue with real-path measurements. This issue supplies those numbers. The trace data covers before and after through real provider turns. It measures the cause, using a workaround in place of that patch. Related: #15124 (waiting state while capture stalls), #23079.
Suggested fix
Keep one checkpoint index per workspace and reuse it, so add -A only rehashes files whose stat data changed. A review of the bundle and tests in a scratch repo found that a naive version is unsafe. Restore deleted a committed file in one test. These guards are needed:
- Reconcile ignored entries after
add -A.add -Anever adds a newly force-added file and never drops a newly ignored one, so a reused index drifts from what a fresh capture would record:- Add HEAD paths missing from the index:
git diff-index --cached --diff-filter=D --name-only -z HEAD | git update-index --add --remove -z --stdin. - Remove ignored entries that HEAD does not track: intersect
ls-files -ci --exclude-standardwithdiff-index --cached --diff-filter=A --name-only HEAD, thenupdate-index --force-remove.
- Add HEAD paths missing from the index:
- Copy, capture, then rename. Copy the saved index to a temp file and backdate it as the user-index copy already is. Capture on the copy and rename it over the saved index only on success. On any failure, delete the saved index and fall back to today's path. A leftover
.lockon a shared saved index would otherwise make every later capture fail. - Fall back to today's path for sparse checkouts and nested workspaces. Skip-worktree bits would otherwise be frozen at seed time. A workspace whose cwd is a subdirectory stages only its subtree.
- Key the saved index by resolved workspace path and store it in the per-worktree git dir, so
worktree removecleans it up. Pass-c core.splitIndex=falseas the other temp-index paths do.
Two smaller changes would cut the lock queueing:
- Don't take the workspace lock in
materializeBaselineCheckpoint; it only reads a ref. - Run the numstat diff and
listAuthoredPathsincaptureafter releasing the lock; they read immutable commits.
Workaround
Commit untracked files locally so they match HEAD.
Activity