Skip to content

fix(batch): write batch_results.json as each case finishes, atomically - #1574

Merged
hakuturu583 merged 1 commit into
masterfrom
fix/issue-1566
Nov 5, 2026
Merged

hakuturu583 merged 1 commit into
masterfrom
fix/issue-1566

Conversation

@hakuturu583

Copy link
Copy Markdown
Collaborator

🐛 Bug Fix

Fixes #1566.

examples/run.py wrote batch_results.json only after run_all() returned, and never after Ctrl-C or SIGTERM. The viewer decides whether a directory is a batch only from that file. So while a glob batch ran, or for ever after an interrupt, the viewer listed it as a single run of its first case: a batch with 2 failures out of 3 showed as "1/1 passed". batch_report.py also wrote the file with a plain write_text, so a reader could see it half-written.

🛠️ Changes

  • atomic_file.py (new, Codon-CHECKED): write_text_atomically(path, text) writes a temp file in the same directory, sets 0o644 and os.replaces it over the target, removing the temp file on any exception. It is factored out of maps.cache._write_atomically, which now calls it.
  • batch_report.py: write_batch_result_json writes atomically. The content and schema are unchanged.
  • scenario_queue.py: run_all(on_result=...), called once per case after its last attempt, including errored cases. The call sits outside the retry try, so it is never counted as a failed attempt.
  • examples/run.py (run_batch): rewrites batch_results.json after each case, and once more in a finally if interrupted, with the cases run so far and in the same schema. A failed progress write is logged, and the batch keeps going. A completed batch gets the same file as before.
  • Tests:
    • test_batch_results_progress.py:
      • a batch interrupted after 2 of 3 cases lists as a batch, 1/2, with no temp file;
      • an interruption before any case writes nothing;
      • an interruption before the rename keeps the old file;
      • the atomic write replaces the file whole.
    • test_scenario_queue.py: on_result is called once per case, including retried and errored cases.

⚠️ Breaking Changes

None. batch_results.json is now always mode 0o644, where it used to follow the umask.

🔍 Review

A fresh-context review passed it. It confirmed:

  • progress files and the final file have the same shape;
  • no viewer reader depends on rerun_errors, which is absent from progress files as it is from a final file with no errors;
  • totals come from the entries, so missing cases are never shown as passed;
  • the temp name never matches the scanner's globs;
  • SLURM runs are unaffected.

Optional points it noted that this PR doesn't take:

  • an errored case can be missing from an interrupted file if the following reset_world() raises;
  • one test raises KeyboardInterrupt where a RuntimeError would prove more.

✅ Testing

  • Codon: scenario-check --library on atomic_file, batch_report, maps.cache and scenario_queue passed (4 checked modules and the 154 they import).
  • Targeted tests: 1452 passed, 2 skipped. Library and sweep tests: 67 passed.
  • pre-commit run --files on the changed files: all hooks pass.

Generated by Claude Code

A local batch wrote batch_results.json only after run_all() returned, so a
batch still running, or cut short by Ctrl-C or a SIGTERM (_Terminated skips
the summary), had no summary file. The viewer takes a directory for a batch
only from that file, and otherwise listed it as a single run of its first
case ("1/1 passed" for a batch with two failures of three).

- ScenarioQueue.run_all() takes on_result, called with each case's index
  and result once its last attempt is done (outside the attempts' try, so
  what it raises is not a failed attempt to retry).
- run_batch() rewrites batch_results.json from it with the cases run so
  far (same schema; a failed write is logged and the batch goes on), and
  once more in its finally when the run was cut short. The final summary
  is unchanged.
- The file is written atomically: a new checked module, atomic_file
  (write_text_atomically: mkstemp in the same directory, then os.replace),
  factored out of maps.cache._write_atomically, which now delegates to it.
  batch_report uses it, so the viewer never reads half a file.

Fixes #1566

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6ipH3q3UpoqgCViXxEJjW
Signed-off-by: Masaya Kataoka <ms.kataoka@gmail.com>
@hakuturu583 hakuturu583 added the bump patch label Nov 5, 2026 — with Claude
@hakuturu583
hakuturu583 merged commit 45f2dfb into master Nov 5, 2026
16 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(batch): a running or interrupted local batch shows in the viewer as a single run of its first case

1 participant