Repository navigation
fix(batch): write batch_results.json as each case finishes, atomically - #1574
Merged
Merged
Conversation
A local batch wrote batch_results.json only after run_all() returned, so a
batch still running, or cut short by Ctrl-C or a SIGTERM (_Terminated skips
the summary), had no summary file. The viewer takes a directory for a batch
only from that file, and otherwise listed it as a single run of its first
case ("1/1 passed" for a batch with two failures of three).
- ScenarioQueue.run_all() takes on_result, called with each case's index
and result once its last attempt is done (outside the attempts' try, so
what it raises is not a failed attempt to retry).
- run_batch() rewrites batch_results.json from it with the cases run so
far (same schema; a failed write is logged and the batch goes on), and
once more in its finally when the run was cut short. The final summary
is unchanged.
- The file is written atomically: a new checked module, atomic_file
(write_text_atomically: mkstemp in the same directory, then os.replace),
factored out of maps.cache._write_atomically, which now delegates to it.
batch_report uses it, so the viewer never reads half a file.
Fixes #1566
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6ipH3q3UpoqgCViXxEJjW
Signed-off-by: Masaya Kataoka <ms.kataoka@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🐛 Bug Fix
Fixes #1566.
examples/run.pywrotebatch_results.jsononly afterrun_all()returned, and never after Ctrl-C or SIGTERM. The viewer decides whether a directory is a batch only from that file. So while a glob batch ran, or for ever after an interrupt, the viewer listed it as a single run of its first case: a batch with 2 failures out of 3 showed as "1/1 passed".batch_report.pyalso wrote the file with a plainwrite_text, so a reader could see it half-written.🛠️ Changes
atomic_file.py(new, Codon-CHECKED):write_text_atomically(path, text)writes a temp file in the same directory, sets 0o644 andos.replaces it over the target, removing the temp file on any exception. It is factored out ofmaps.cache._write_atomically, which now calls it.batch_report.py:write_batch_result_jsonwrites atomically. The content and schema are unchanged.scenario_queue.py:run_all(on_result=...), called once per case after its last attempt, including errored cases. The call sits outside the retrytry, so it is never counted as a failed attempt.examples/run.py(run_batch): rewritesbatch_results.jsonafter each case, and once more in afinallyif interrupted, with the cases run so far and in the same schema. A failed progress write is logged, and the batch keeps going. A completed batch gets the same file as before.test_batch_results_progress.py:test_scenario_queue.py:on_resultis called once per case, including retried and errored cases.None.
batch_results.jsonis now always mode 0o644, where it used to follow the umask.🔍 Review
A fresh-context review passed it. It confirmed:
rerun_errors, which is absent from progress files as it is from a final file with no errors;Optional points it noted that this PR doesn't take:
reset_world()raises;KeyboardInterruptwhere aRuntimeErrorwould prove more.✅ Testing
scenario-check --libraryonatomic_file,batch_report,maps.cacheandscenario_queuepassed (4 checked modules and the 154 they import).pre-commit run --fileson the changed files: all hooks pass.Generated by Claude Code