Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions docs/src/reference/experimental/autoharness.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ works in three phases; click a box to jump to its section:
│ • one monomorphization only │ │ codegen → CBMC → summary │
│ • no aliasing between args (#4750) │ │ (see the architecture diagram below) │
│ • bounded args only up to bound │ └──────────────────────────────────────────┘
│ • ctor may be vacuous (#4757) │
│ • ctor may reach only some inputs │
│ • mined-invariant heuristic (#4763) │
└───────────────────────────────────────┘
</code></pre>
Expand Down Expand Up @@ -737,9 +737,17 @@ This applies to harnesses marked "(ctor)", i.e. runs with `--constructor-args` (
[Constructor-based generation](#constructor-based-generation)). If the chosen
constructor is *unsatisfiable* for the generated type — an assert-guarded constructor every
argument of which trips an assertion, or a checked constructor that always returns
`None`/`Err` — the generated body assumes `false` on all paths and the harness becomes
**vacuous**, reporting `Success` without checking anything. Kani does not yet detect this case;
see [#4757](https://github.com/model-checking/kani/issues/4757).
`None`/`Err` — the generated body assumes `false` on all paths. Mined conditions that no
generated value satisfies have the same effect. If no generated input then reaches the function
under test, the harness is **vacuous**: none of its checks can fail. Kani covers the call to that
function (`` autoharness: a generated input reaches `...` ``) and reports such a harness as a
failure, with the line `` Vacuous harness: no generated input reaches `...` ``. For a function
with a contract, its `requires` clause is assumed inside the call, after the cover, so a
precondition that excludes every generated input is not reported as vacuous. A harness that
reaches the function with only some of its inputs, for example with `None` for an `Option`
argument whose payload cannot be generated, is reported as usual; the excluded inputs, such as
the `Some` path, are never checked. `--constructor-args` cannot be combined with
`--output-format=old`, which cannot detect a vacuous harness.

### Mined-invariant heuristic

Expand Down
49 changes: 44 additions & 5 deletions kani-compiler/src/kani_middle/transform/automatic.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1659,11 +1659,6 @@ impl AutomaticArbitraryPass {
/// arguments and panic paths converted into assumptions
/// (c.f. [find_unchecked_constructor][crate::kani_middle::find_unchecked_constructor]).
/// Returns None if the constructor body contains constructs the inliner does not support.
///
/// Soundness caveat: if the constructor is unsatisfiable for `ty` (every argument trips an
/// assertion), the generated body assumes `false` on all paths and the harness becomes
/// vacuous, reporting Success without checking anything. Not yet detected; tracked in
/// <https://github.com/model-checking/kani/issues/4757>.
fn generate_unchecked_ctor_body(
&self,
tcx: TyCtxt,
Expand Down Expand Up @@ -2024,6 +2019,8 @@ pub struct AutomaticHarnessPass {
init_contracts_hook: Instance,
reset_clause_depth: Instance,
kani_autoharness_intrinsic: FnDef,
/// The FnDef of KaniHook::Cover, c.f. `cover_function_reached`.
kani_cover: FnDef,
/// Whether --check-invariants is enabled: check values returned by verified functions
/// against the type's mined invariant conjuncts.
check_invariants: bool,
Expand All @@ -2044,16 +2041,55 @@ impl AutomaticHarnessPass {
*kani_fns.get(&KaniModel::ResetContractClauseDepth.into()).unwrap();
let reset_clause_depth =
Instance::resolve(reset_clause_depth, &GenericArgs(vec![])).unwrap();
let kani_cover = *kani_fns.get(&KaniHook::Cover.into()).unwrap();
let check_invariants = query_db.args().autoharness_check_invariants;
Self {
models: AnyModels::new(query_db),
check_fmt_models,
init_contracts_hook,
reset_clause_depth,
kani_autoharness_intrinsic,
kani_cover,
check_invariants,
}
}

/// Under --constructor-args, emit `kani::cover(true, msg)` at `source`, where the harness is
/// about to call `function` (directly, or through a formatting model). The assumptions made
/// while generating the arguments (a constructor that never returns, mined conditions that
/// contradict it or each other) can exclude every input, so that the harness checks nothing;
/// the driver then reports the harness as vacuous
/// (<https://github.com/model-checking/kani/issues/4757>).
fn cover_function_reached(
&self,
body: &mut MutableBody,
source: &mut SourceInstruction,
function: Instance,
) {
if !self.models.constructor_args {
return;
}
let cover_inst = Instance::resolve(self.kani_cover, &GenericArgs(vec![]))
.expect("the cover hook is always registered");
let span = source.span(body.blocks());
let true_op = Operand::Constant(ConstOperand {
span,
user_ty: None,
const_: MirConst::from_bool(true),
});
// kani-driver recognizes this cover by the start of its description.
let msg =
format!("{}`{}`", kani_metadata::AUTOHARNESS_VACUITY_COVER_PREFIX, function.name());
let msg_op = body.new_str_operand(&msg, span);
let unit_lcl = body.new_local(Ty::new_tuple(&[]), span, Mutability::Not);
body.insert_call(
&cover_inst,
source,
InsertPosition::Before,
vec![true_op, msg_op],
Place::from(unit_lcl),
);
}
}

impl TransformPass for AutomaticHarnessPass {
Expand Down Expand Up @@ -2121,6 +2157,7 @@ impl TransformPass for AutomaticHarnessPass {
source.span(harness_body.blocks()),
Mutability::Not,
);
self.cover_function_reached(&mut harness_body, &mut source, fn_to_verify);
harness_body.insert_call(
&model_inst,
&mut source,
Expand Down Expand Up @@ -2202,6 +2239,8 @@ impl TransformPass for AutomaticHarnessPass {
);
let ret_place = Place::from(ret_lcl);

self.cover_function_reached(&mut harness_body, &mut source, fn_to_verify);

// Call `fn_to_verify` on the nondeterministic arguments generated above.
harness_body.insert_call(
&fn_to_verify,
Expand Down
51 changes: 50 additions & 1 deletion kani-driver/src/args/autoharness_args.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
use std::path::PathBuf;

use crate::args::list_args::Format;
use crate::args::{ValidateArgs, VerificationArgs, validate_std_path};
use crate::args::{OutputFormat, ValidateArgs, VerificationArgs, validate_std_path};
use crate::util::warning;
use clap::{Error, Parser, error::ErrorKind};
use kani_metadata::{
Expand Down Expand Up @@ -244,6 +244,16 @@ impl ValidateArgs for CargoAutoharnessArgs {
));
}

if self.common_autoharness_args.constructor_args
&& self.verify_opts.output_format() == OutputFormat::Old
{
return Err(Error::raw(
ErrorKind::ArgumentConflict,
"Conflicting options: --constructor-args isn't compatible with \
--output-format=old, since `old` cannot detect a vacuous harness.",
));
}
Comment thread
feliperodri marked this conversation as resolved.
Resolved

Ok(())
}
}
Expand Down Expand Up @@ -296,6 +306,16 @@ impl ValidateArgs for StandaloneAutoharnessArgs {
));
}

if self.common_autoharness_args.constructor_args
&& self.verify_opts.output_format() == OutputFormat::Old
{
return Err(Error::raw(
ErrorKind::ArgumentConflict,
"Conflicting options: --constructor-args isn't compatible with \
--output-format=old, since `old` cannot detect a vacuous harness.",
));
}

Ok(())
}
}
Expand Down Expand Up @@ -352,4 +372,33 @@ mod tests {
assert_eq!(args.string_bound, AUTOHARNESS_STR_BOUND);
assert_eq!(args.bounded_arbitrary_bound, AUTOHARNESS_BOUNDED_ARBITRARY_BOUND);
}

/// Both subcommand forms reject `--constructor-args` with `--output-format=old`, and accept
/// either option on its own.
#[test]
fn constructor_args_reject_old_output_format() {
// `validate` also checks the standalone input is a regular file, so use a real one.
let dir = tempfile::tempdir().unwrap();
let input = dir.path().join("input.rs");
std::fs::write(&input, "").unwrap();
let validate = |options: &[&str]| {
let mut args = vec!["autoharness", "-Z", "autoharness"];
args.extend_from_slice(options);
let cargo = CargoAutoharnessArgs::try_parse_from(&args).unwrap().validate();
args.push(input.to_str().unwrap());
let standalone = StandaloneAutoharnessArgs::try_parse_from(&args).unwrap().validate();
[cargo, standalone]
};

for result in validate(&["--constructor-args", "--output-format=old"]) {
let err = result.expect_err("expected --constructor-args to reject old output");
assert_eq!(err.kind(), ErrorKind::ArgumentConflict);
assert!(err.to_string().contains("--constructor-args"), "{err}");
}
for options in [["--constructor-args"], ["--output-format=old"]] {
for result in validate(&options) {
assert!(result.is_ok(), "{options:?}: {:?}", result.err());
}
}
}
}
9 changes: 8 additions & 1 deletion kani-driver/src/call_cbmc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ use crate::args::{OutputFormat, VerificationArgs};
use crate::cbmc_output_parser::{
CheckStatus, ParserItem, Property, VerificationOutput, extract_results, process_cbmc_output,
};
use crate::cbmc_property_renderer::{format_coverage, format_result, kani_cbmc_output_filter};
use crate::cbmc_property_renderer::{
format_coverage, format_result, is_vacuous_harness_cover, kani_cbmc_output_filter,
};
use crate::coverage::cov_results::{CoverageCheck, CoverageResults};
use crate::coverage::cov_results::{CoverageRegion, CoverageTerm};
use crate::session::KaniSession;
Expand Down Expand Up @@ -498,6 +500,11 @@ impl VerificationResult {
{
let (mut status, mut failed_properties) =
verification_outcome_from_properties(&results, should_panic);
// `format_result` prints the explanation.
if results.iter().any(is_vacuous_harness_cover) {
status = VerificationStatus::Failure;
failed_properties = FailedProperties::Other;
}
// A dropped quantifier makes the analysis unsound: a `kani::assume` containing one is
// silently not enforced, so a "successful" result may be vacuous. Kani must never
// report success in that case -- force a failure and (via the rendered error) direct
Expand Down
24 changes: 23 additions & 1 deletion kani-driver/src/cbmc_property_renderer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ const UNSUPPORTED_CONSTRUCT_DESC: &str = "is not currently supported by Kani";
const UNWINDING_ASSERT_DESC: &str = "unwinding assertion loop";
const UNWINDING_ASSERT_REC_DESC: &str = "recursion unwinding assertion";
const UNDEFINED_FUNCTION_DESC: &str = "undefined function should be unreachable";
const VACUITY_COVER_DESC: &str = kani_metadata::AUTOHARNESS_VACUITY_COVER_PREFIX;

impl ParserItem {
/// Determines if an item must be skipped or not.
Expand Down Expand Up @@ -249,7 +250,9 @@ fn format_item_terse(_item: &ParserItem) -> Option<String> {
/// This function reports the results of normal checks (e.g. assertions and
/// arithmetic overflow checks) and cover properties (specified using the
/// `kani::cover` macro) separately. Cover properties currently do not impact
/// the overall verification success or failure.
/// the overall verification success or failure, except that a harness fails
/// when no input reaches its autoharness vacuity cover (see
/// `is_vacuous_harness_cover`).
///
/// TODO: We could `write!` to `result_str` instead
/// <https://github.com/model-checking/kani/issues/1480>
Expand Down Expand Up @@ -389,6 +392,10 @@ pub fn format_result(
let failure_message = build_failure_message(prop.description.clone(), &prop.trace.clone());
result_str.push_str(&failure_message);
}
for message in properties.iter().filter_map(vacuity_message) {
let vacuity_line = format!("{message}\n");
result_str.push_str(&vacuity_line);
}

let verification_result = if status == VerificationStatus::Success {
style("SUCCESSFUL").green()
Expand Down Expand Up @@ -550,6 +557,21 @@ pub fn postprocess_result(properties: Vec<Property>, extra_ptr_checks: bool) ->
update_results_of_cover_checks(results_after_code_coverage)
}

/// Whether `prop` is a `cover_function_reached` cover (kani-compiler, `automatic.rs`) that no
/// input can satisfy, which makes the harness vacuous.
pub fn is_vacuous_harness_cover(prop: &Property) -> bool {
prop.is_cover_property()
&& prop.description.starts_with(VACUITY_COVER_DESC)
&& matches!(prop.status, CheckStatus::Unsatisfiable | CheckStatus::Unreachable)
}

/// The line reporting a vacuous harness, if `prop` is a cover that makes it vacuous.
pub fn vacuity_message(prop: &Property) -> Option<String> {
let function = prop.description.strip_prefix(VACUITY_COVER_DESC)?;
is_vacuous_harness_cover(prop)
.then(|| format!("Vacuous harness: no generated input reaches {function}"))
}

/// Determines if there is property with status `FAILURE` and the given description
fn has_check_failure(properties: &Vec<Property>, description: &str) -> bool {
for prop in properties {
Expand Down
53 changes: 33 additions & 20 deletions kani-driver/src/sarif.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@

use crate::call_cbmc::ExitStatus;
use crate::cbmc_output_parser::{CheckStatus, Property, SourceLocation, TraceItem};
use crate::cbmc_property_renderer::vacuity_message;
use crate::harness_runner::HarnessResult;
use crate::session::KaniSession;
use anyhow::{Context, Result};
use kani_metadata::HarnessMetadata;
use pathdiff::diff_paths;
use serde::Serialize;
use std::collections::BTreeMap;
Expand Down Expand Up @@ -180,6 +182,15 @@ impl SarifLog {
},
});
}
for text in properties.iter().filter_map(vacuity_message) {
let rule_id = "kani.autoharness.vacuous";
let desc = "No generated input reaches the function under test";
rules.entry(rule_id.to_string()).or_insert_with(|| ReportingDescriptor {
id: rule_id.to_string(),
short_description: Message { text: desc.to_string() },
});
sarif_results.push(harness_level_result(harness, rule_id, &text));
}
}
Err(exit_status) => {
let (rule_id, desc) = exit_status_rule(exit_status);
Expand All @@ -188,26 +199,7 @@ impl SarifLog {
short_description: Message { text: desc.to_string() },
});

sarif_results.push(SarifResult {
rule_id: rule_id.to_string(),
level: "error",
message: Message { text: format!("[{}] {desc}", harness.pretty_name) },
locations: vec![Location {
physical_location: PhysicalLocation {
artifact_location: ArtifactLocation {
uri: relativize_path(&harness.original_file),
},
region: Region {
start_line: harness.original_start_line as u32,
start_column: None,
},
},
}],
properties: ResultProperties {
harness: harness.pretty_name.clone(),
property_name: None,
},
});
sarif_results.push(harness_level_result(harness, rule_id, desc));
}
}
}
Expand All @@ -230,6 +222,27 @@ impl SarifLog {
}
}

/// An error result about `harness` as a whole, located at the harness's own source.
fn harness_level_result(harness: &HarnessMetadata, rule_id: &str, text: &str) -> SarifResult {
SarifResult {
rule_id: rule_id.to_string(),
level: "error",
message: Message { text: format!("[{}] {text}", harness.pretty_name) },
locations: vec![Location {
physical_location: PhysicalLocation {
artifact_location: ArtifactLocation {
uri: relativize_path(&harness.original_file),
},
region: Region {
start_line: harness.original_start_line as u32,
start_column: None,
},
},
}],
properties: ResultProperties { harness: harness.pretty_name.clone(), property_name: None },
}
}

fn sarif_level(status: &CheckStatus) -> Option<&'static str> {
match status {
CheckStatus::Failure => Some("error"),
Expand Down
4 changes: 4 additions & 0 deletions kani_metadata/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ pub const AUTOHARNESS_STR_BOUND: u64 = 4;
/// makes simple `String` harnesses exceed Kani's default 60s harness timeout.
pub const AUTOHARNESS_BOUNDED_ARBITRARY_BOUND: u64 = 4;

/// The start of the description of the cover that automatic harnesses place before the call to
/// the function under test. kani-driver reports a harness whose cover no input reaches as vacuous.
pub const AUTOHARNESS_VACUITY_COVER_PREFIX: &str = "autoharness: a generated input reaches ";

/// The structure of `.kani-metadata.json` files, which are emitted for each crate
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct KaniMetadata {
Expand Down
Loading
Loading