Skip to content

Autoharness: report bounded Rc/Arc reference-count models - #5070

Open
acearyanarun wants to merge 5 commits into
model-checking:mainfrom
acearyanarun:autoharness-refcount-bounded-reporting
Open

acearyanarun wants to merge 5 commits into
model-checking:mainfrom
acearyanarun:autoharness-refcount-bounded-reporting

Conversation

@acearyanarun

@acearyanarun acearyanarun commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Draft — Local validation complete; CI and maintainer review pending. The pinned nightly toolchain was installed successfully on macOS. The compiler build, unit tests, regression tests, formatting, and Clippy checks all passed locally. This PR contains only the independent reporting change. Shared-state tests requiring #4788 remain separate.

Summary

AutoHarness-generated Rc/Arc values have finitely modeled reference counts, but current AutoHarness output describes these harnesses as verifying "all possible inputs" without identifying that limitation. This PR makes the limitation visible without disabling default Rc/Arc support.

  • Adds an independent HarnessMetadata::is_refcount_bounded caveat.
  • Marks affected harnesses (refcount) in the AutoHarness summary, adds an explanatory note, and qualifies the per-function progress message.
  • Exposes is_refcount_bounded in --export-json, including schema, fixtures, and RFC documentation updates.
  • Adds tests for the marker, metadata compatibility, JSON reporting, type nesting, and non-vacuous uniquely owned states.
  • Documents the finite-model limitation and remaining caveats.

This PR does not change any Rc/Arc generator.

Why this matters

Today, generated Rc/Arc values are uniquely owned (strong_count == 1, weak_count == 0). PR #4788 proposes additional shared-ownership states (strong_count in {1, 2} and weak_count in {0, 1}). Neither model covers all exact count values a caller may supply, so a successful verification result needs to communicate that limitation. A function checking, for example, Rc::strong_count(&x) > 2 can still depend on unmodeled states.

The existing (bounded) marker is gated by --bounded-arguments; reusing it would prevent default generation of Rc/Arc arguments. The new (refcount) caveat is separate and non-gating.

Implementation

  • The compiler inspects generated argument types, following references, arrays, tuples, ADT fields and generic arguments, while skipping PhantomData. It deliberately errs toward warning on uncertain nesting rather than silently omitting a caveat.
  • The metadata field defaults to false when absent in older serialized metadata.
  • The driver adds a summary marker and note, plus an explicit qualification when printing individual verification progress.
  • JSON export includes the field, and the schema and fixtures are updated.
  • Regression tests check direct and nested types, the AnyRc/AnyArc reporting paths, negative controls, and coverage expectations for uniquely owned states.

The type-based detection may flag types whose generation cannot actually produce an Rc/Arc. The warning does not claim complete modeling. It also does not cover manual harnesses using kani::any().

Difference from #4788

#4788 This PR
Focus Generate additional Rc/Arc ownership states Report the finite modeling limitation
Code Rc/Arc arbitrary-value generators Compiler metadata, driver output, JSON export, docs
Dependency Core generator change Independent of #4788

Shared-state regression cases that depend on #4788 are intentionally excluded from this PR; they can be contributed when the generator change lands.

Testing

Passed locally on macOS using nightly-2026-10-23:

  • cargo build-dev — passed, including kani-compiler.
  • cargo test -p kani-driver -p kani_metadata — 214 driver tests and 4 metadata tests passed; 2 documentation tests passed, 1 ignored.
  • cargo run -p compiletest -- --suite json-handler --mode exec autoharness-refcount — 1 passed, 0 failed.
  • cargo run -p compiletest -- --suite script-based-pre --mode exec --force-rerun cargo_autoharness_smart_pointers — 1 passed, 0 failed.
  • cargo run -p compiletest -- --suite script-based-pre --mode exec --force-rerun autoharness — 42 passed, 0 failed.
  • ./scripts/kani-fmt.sh --check — passed.
  • cargo clippy --workspace --tests -- -D warnings — passed.
  • RUSTFLAGS="--cfg=kani_sysroot" cargo clippy --workspace -- -D warnings — passed.

The broader AutoHarness suite includes the smart-pointer test, so these counts overlap.

Pending: GitHub CI and maintainer review. The shared-ownership regression tests depending on #4788 are intentionally excluded.

Related work

Part of #4752. Related: #4788 (generator), #4958 (aliasing metadata), and #4991 (AutoHarness documentation restructure; may require rebase/conflict resolution).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WddFow1MppHhSFwg9HdZCz

Kani models the reference counts of generated Rc/Arc values with only
finitely many states (model-checking#4752), yet autoharness reported such harnesses as
checked "against all possible inputs" with no caveat, since Rc/Arc
arguments are unbounded in the pointee and need no --bounded-arguments.

Add an independent, non-gating caveat, mirroring is_ctor_based:
- HarnessMetadata::is_refcount_bounded, set when some generated value
  contains an Rc/Arc, found by walking the argument types (references,
  raw pointers, arrays, slices, tuples, pattern types, ADT fields and
  generic arguments; PhantomData skipped; over-approximating, so a
  missed caveat is not possible). Formatting-trait harnesses use the
  impl's self type.
- A "(refcount)" marker in the summary table (it fits the Kind column,
  so other rows do not shift), a note after the table, and a qualified
  "all possible inputs" message.
- is_refcount_bounded in the --export-json document, with the schema,
  fixtures, checker and RFC 0015 updated.

ArgSupport::Bounded is deliberately not reused: it is gated behind
--bounded-arguments, which would disable Rc/Arc support by default.
The Rc/Arc generators are unchanged (see model-checking#4788).

Tests: a json-handler test asserting the export field, marker, note and
message per function (direct, AnyRc/AnyArc model paths, nested field,
tuple, reference, Box; negative controls incl. PhantomData), with
covers that must be SATISFIED for the generated unique state; updated
smart-pointer expectations; driver unit tests.

Docs: limitations of the finite model in the autoharness reference.

Part of model-checking#4752.
@github-actions github-actions Bot added Z-EndToEndBenchCI Tag a PR to run benchmark CI Z-CompilerBenchCI Tag a PR to run benchmark CI labels Oct 27, 2026
@acearyanarun
acearyanarun marked this pull request as ready for review October 27, 2026 00:54
@acearyanarun
acearyanarun requested review from a team as code owners October 27, 2026 00:54
@acearyanarun

Copy link
Copy Markdown
Contributor Author

@wodex1nhaoIeng @srivatsansamraj @Tianshu-Huang @CYJ904 Hi team! I've opened PR #5070 to address the bounded Rc/Arc reference-count reporting limitation in AutoHarness (#4752).

The changes add a separate (refcount) marker to verification summaries and JSON metadata without modifying the existing Rc/Arc generators or depending on #4788.

Local builds, unit tests, AutoHarness regression tests, formatting, and Clippy checks have all passed.

Could you review the changes when you get a chance, particularly the recursive type detection, metadata compatibility, and reporting behavior?

@feliperodri feliperodri added [C] Bug This is a bug. Something isn't working. [F] Soundness Kani failed to detect an issue Z-Autoharness Issue related to autoharness subcommand labels Nov 6, 2026
@feliperodri feliperodri added this to the Autoharness milestone Nov 6, 2026
@feliperodri
feliperodri requested a balanced review from Copilot November 6, 2026 19:10
@feliperodri feliperodri self-assigned this Nov 6, 2026
@feliperodri

Copy link
Copy Markdown
Member

@acearyanarun could you resolve the branch conflicts?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues
What changed in this PR

Adds explicit reporting that AutoHarness-generated Rc/Arc inputs have finitely modeled reference counts, surfacing this limitation in CLI output and JSON export without changing generation behavior.

Changes:

  • Introduces HarnessMetadata::is_refcount_bounded and propagates it through compiler metadata, driver output, and JSON export/schema.
  • Updates AutoHarness summary/progress messaging to mark affected harnesses with (refcount) and print an explanatory note.
  • Adds regression tests/fixtures and documentation covering the new caveat and export field.
File Description
tests/​script-based-pre/​cargo_autoharness_smart_pointers/​src/​lib.rs Updates test description to note (refcount) marking for Rc/Arc.
tests/​script-based-pre/​cargo_autoharness_smart_pointers/​smart-pointers.expected Updates expected AutoHarness summary rows and adds the (refcount) note.
tests/​json-handler/​schema-validation/​kani_json_schema.json Extends schema with is_refcount_bounded.
tests/​json-handler/​schema-validation/​fixture-non-completed-outcomes.json Adds is_refcount_bounded to fixtures.
tests/​json-handler/​schema-validation/​fixture-completed-failure.json Adds is_refcount_bounded to fixtures.
tests/​json-handler/​schema-validation/​check_fixtures.py Adds negative fixture mutations for is_refcount_bounded.
tests/​json-handler/​autoharness-refcount/​test.sh New end-to-end JSON/log assertions for (refcount) reporting.
tests/​json-handler/​autoharness-refcount/​test.rs New functions covering direct/nested Rc/Arc detection and non-vacuity.
tests/​json-handler/​autoharness-refcount/​config.yml Adds script-based test configuration.
rfc/​src/​rfcs/​0015-export-json.md Documents the new JSON field and semantics.
kani_metadata/​src/​harness.rs Adds HarnessMetadata::is_refcount_bounded with serde default.
kani-driver/​src/​sarif.rs Updates test harness metadata construction with new field.
kani-driver/​src/​metadata.rs Updates test harness metadata construction with new field.
kani-driver/​src/​harness_runner.rs Qualifies per-function “all possible inputs” messaging using the new caveat.
kani-driver/​src/​frontend/​schema_utils.rs Exports is_refcount_bounded to JSON and adds test assertions.
kani-driver/​src/​autoharness/​mod.rs Adds (refcount) marker to summary rows and prints explanatory note.
kani-compiler/​src/​kani_middle/​mod.rs Adds type-walk helper contains_refcounted_ptr for Rc/Arc detection.
kani-compiler/​src/​kani_middle/​metadata.rs Threads is_refcount_bounded through automatic harness metadata generation.
kani-compiler/​src/​kani_middle/​codegen_units.rs Computes is_refcount_bounded for autoharness caveats during partitioning.
docs/​src/​reference/​experimental/​autoharness.md Documents the finite reference-count modeling caveat and reporting.

Review effort: Lite

Comment thread kani-driver/src/harness_runner.rs Outdated
Comment on lines 100 to 116
/// Identify the worker on every line of a rendered verification result.
/// What an automatic harness checks its function against, for the "Autoharness: Checking
/// function ..." message. A harness with a caveat does not check *all* possible inputs, so
/// qualify the message accordingly so it does not masquerade as exhaustive (c.f. the
/// "(bounded)" and "(refcount)" markers in the summary).
fn autoharness_inputs_description(harness: &HarnessMetadata) -> &'static str {
match (harness.is_bounded, harness.is_refcount_bounded) {
(false, false) => "all possible inputs",
(true, false) => "all possible inputs (bounded for some arguments)",
(false, true) => "all possible inputs (with finitely modeled Rc/Arc reference counts)",
(true, true) => {
"all possible inputs (bounded for some arguments, with finitely modeled Rc/Arc reference counts)"
}
}
}

fn prefix_thread_lines(output: &str, thread_index: usize) -> String {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d02e67b: the doc comment is back on prefix_thread_lines, and autoharness_inputs_description has its own.

Comment thread kani_metadata/src/harness.rs Outdated
Comment on lines +58 to +63
/// representatives. Its verification result therefore does not cover every reference
/// count a caller could produce, even though `Rc`/`Arc` arguments need no
/// `--bounded-arguments` and do not set `is_bounded`
/// (<https://github.com/model-checking/kani/issues/4752>).
#[serde(default)]
pub is_refcount_bounded: bool,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to has_refcount_caveat in d02e67b, everywhere it appears: HarnessMetadata, the --export-json field, the JSON schema and its fixtures, RFC 0015, and the tests. I didn't keep the old JSON name with #[serde(rename)], since the field hasn't been released yet.

Comment thread kani-compiler/src/kani_middle/mod.rs Outdated
/// that the only failure mode, the walk reports `true` if it exceeds its budget (e.g. for
/// polymorphically recursive types). Values not visible in the type (behind `dyn Trait`, or
/// function pointers) cannot be generated by automatic harnesses and are not considered.
pub fn contains_refcounted_ptr(tcx: TyCtxt, ty: Ty, visited: &mut FxHashSet<Ty>) -> bool {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now pub(crate) (d02e67b). It's only used inside kani-compiler.

@feliperodri feliperodri left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed description. The reporting change is clear and the tests cover the cases it claims. Needs changes before merge:

  1. Merge conflict. The PR is not mergeable (state dirty). Please rebase on main.
  2. Draft note. The description starts with "Draft — local validation complete", but the PR isn't a draft. Remove the note, or convert the PR to a draft.
  3. Copilot's points are valid. Please fix the doc comment at harness_runner.rs:100 (it now sits on the wrong function). Consider renaming is_refcount_bounded to something like has_refcount_caveat, so it isn't read as related to --bounded-arguments. Make contains_refcounted_ptr pub(crate) unless it's meant as public API.
  4. Checks. Three macOS jobs are cancelled. Please re-run them.

The [F] Soundness label fits: the PR makes the overclaim visible (a bounded model reported as "all possible inputs").

@feliperodri feliperodri removed their assignment Nov 7, 2026
The autoharness reference page was restructured on main (model-checking#4991). Move this
branch's reference-count documentation into the new structure: a sentence
under "Box, Rc and Arc", a "(refcount)" entry under "Summary table and
markers", and a "Reference counts" subsection under "Soundness caveats".
- Rename `is_refcount_bounded` to `has_refcount_caveat` in the metadata,
  the `--export-json` schema, fixtures, RFC 0015 and tests, so the field is
  not read as related to `--bounded-arguments`.
- Move the `prefix_thread_lines` doc comment back onto that function.
- Make `contains_refcounted_ptr` `pub(crate)`.
- Add the field to `fixture-completed-success.json`, which model-checking#5154 added on
  `main`; the schema requires it.
…aveat

The relocated bullets described a model with shared representatives (from
the closed model-checking#4788) and the overflow trade-off of that design. Generated
Rc/Arc values are always uniquely owned, so say that only the counts 1 and
0 are checked and that no other holder exists.
@acearyanarun
acearyanarun force-pushed the autoharness-refcount-bounded-reporting branch from 1994a45 to ca831cc Compare November 17, 2026 01:24
@acearyanarun

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @feliperodri. I've addressed each point:

  • Conflict: merged main (0af3c25) rather than rebasing. The refcount docs now fit the restructured autoharness page (under "Box, Rc and Arc", "Summary table and markers" and "Soundness caveats → Reference counts"). ca831cc corrects them to the current model: generated Rc/Arc values are always uniquely owned, since Generate nondeterministic reference-count state for Rc/Arc Arbitrary #4788 was closed. I also force-pushed once to tidy the first commit's message; the code is unchanged.
  • Draft note: removed from the description.
  • Copilot's points (d02e67b): the doc comment is back on prefix_thread_lines; is_refcount_bounded is renamed to has_refcount_caveat across the metadata, --export-json, schema, fixtures, RFC 0015 and tests; and contains_refcounted_ptr is pub(crate). The same commit adds the field to fixture-completed-success.json from Add end-to-end tests for the RFC 0015 export #5154, which the schema requires.

Local results (macOS, nightly-2026-11-11, main at a3c8b86 merged): cargo build-dev, cargo test -p kani-compiler (48), cargo test -p kani-driver -p kani_metadata (221 + 4), the json-handler suite (18, including schema and fixture validation), cargo_autoharness_smart_pointers, the autoharness suite (48), kani-fmt --check and both CI clippy commands all pass.

GitHub CI: the new workflow runs are waiting for maintainer approval, so they haven't run yet. Could you approve them when you have a moment?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[C] Bug This is a bug. Something isn't working. [F] Soundness Kani failed to detect an issue Z-Autoharness Issue related to autoharness subcommand Z-CompilerBenchCI Tag a PR to run benchmark CI Z-EndToEndBenchCI Tag a PR to run benchmark CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants