Repository navigation
Report a goto-instrument failure as a failed harness instead of aborting the run - #5167
Open
sankalpsthakur wants to merge 1 commit into
Open
sankalpsthakur wants to merge 1 commit into
sankalpsthakur wants to merge 1 commit into
Conversation
…ing the run When preparing one harness's goto model failed (goto-cc specialization, goto-instrument, or goto-synthesizer, e.g. goto-instrument running out of memory), the error propagated out of the harness runner and the whole run stopped: the results of the other harnesses, the summary, and the autoharness table were lost. Such a failure now fails that harness only. It becomes a failed VerificationResult with a new ExitStatus::InstrumentationFailed variant that carries the error, is printed with the harness name, counts as a failure in the summary and the exit code, and stops the run under --fail-fast like any other failure. The JSON export reports it as a CRASHED outcome with the error as its message, and SARIF as kani.instrumentation.failed. Resolves model-checking#4966
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When preparing one harness's goto model fails (goto-cc, goto-instrument or goto-synthesizer, for example goto-instrument running out of memory), the error aborts the whole run. The results of the other harnesses, the summary and the autoharness table are lost.
With this change that harness is reported as failed, with the error, and the run goes on. The run still exits 1, and
--fail-faststops on it like any other failure. The JSON export reports it asCRASHEDwith the error as its message; SARIF useskani.instrumentation.failed.Behavior change: a failure that affects every harness, such as a missing goto-instrument, now gives one failure per harness instead of one abort. The out-of-memory wording suggested in the issue is not part of this change; goto-instrument's own "Out of memory" output is still printed. With
--concrete-playback, a run whose only failures are of this kind prints the existing "file a bug report" message, as timeouts already do.Testing
call_cbmc.rs,sarif.rsandschema_utils.rs(cargo test -p kani-driver), andcargo check -p kani-driver --tests.goto_instrument_failure_keeps_others. A fakegoto-instrumentfails one of two harnesses; the test checks the summary (1 passed, 1 failed, 2 total), the non-zero exit, and that--fail-faststops after the failure. I have not run it locally, it needs a full Kani build.AI tools assisted with this change; I reviewed it.
Resolves #4966
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.