Skip to content

Report a goto-instrument failure as a failed harness instead of aborting the run - #5167

Open
sankalpsthakur wants to merge 1 commit into
model-checking:mainfrom
sankalpsthakur:fix/4966-continue-after-instrument-failure
Open

sankalpsthakur wants to merge 1 commit into
model-checking:mainfrom
sankalpsthakur:fix/4966-continue-after-instrument-failure

Conversation

@sankalpsthakur

Copy link
Copy Markdown
Contributor

When preparing one harness's goto model fails (goto-cc, goto-instrument or goto-synthesizer, for example goto-instrument running out of memory), the error aborts the whole run. The results of the other harnesses, the summary and the autoharness table are lost.

With this change that harness is reported as failed, with the error, and the run goes on. The run still exits 1, and --fail-fast stops on it like any other failure. The JSON export reports it as CRASHED with the error as its message; SARIF uses kani.instrumentation.failed.

Behavior change: a failure that affects every harness, such as a missing goto-instrument, now gives one failure per harness instead of one abort. The out-of-memory wording suggested in the issue is not part of this change; goto-instrument's own "Out of memory" output is still printed. With --concrete-playback, a run whose only failures are of this kind prints the existing "file a bug report" message, as timeouts already do.

Testing

  • Unit tests in call_cbmc.rs, sarif.rs and schema_utils.rs (cargo test -p kani-driver), and cargo check -p kani-driver --tests.
  • New script test goto_instrument_failure_keeps_others. A fake goto-instrument fails one of two harnesses; the test checks the summary (1 passed, 1 failed, 2 total), the non-zero exit, and that --fail-fast stops after the failure. I have not run it locally, it needs a full Kani build.

AI tools assisted with this change; I reviewed it.

Resolves #4966

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

…ing the run

When preparing one harness's goto model failed (goto-cc specialization,
goto-instrument, or goto-synthesizer, e.g. goto-instrument running out of
memory), the error propagated out of the harness runner and the whole run
stopped: the results of the other harnesses, the summary, and the
autoharness table were lost.

Such a failure now fails that harness only. It becomes a failed
VerificationResult with a new ExitStatus::InstrumentationFailed variant
that carries the error, is printed with the harness name, counts as a
failure in the summary and the exit code, and stops the run under
--fail-fast like any other failure. The JSON export reports it as a
CRASHED outcome with the error as its message, and SARIF as
kani.instrumentation.failed.

Resolves model-checking#4966
@sankalpsthakur
sankalpsthakur requested review from a team as code owners November 13, 2026 14:29
@github-actions github-actions Bot added the Z-EndToEndBenchCI Tag a PR to run benchmark CI label Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Z-EndToEndBenchCI Tag a PR to run benchmark CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

One harness's goto-instrument failure aborts the whole run and discards every other harness's results

1 participant