Skip to content

Migrate --export-json to the RFC 0015 document shape - #4941

Merged
feliperodri merged 10 commits into
model-checking:mainfrom
ivmat:export-json-rfc0015-writer
Oct 21, 2026
Merged

feliperodri merged 10 commits into
model-checking:mainfrom
ivmat:export-json-rfc0015-writer

Conversation

@ivmat

@ivmat ivmat commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Part of #942 (RFC 0015, #4727).

Description

Replaces the ad-hoc --export-json JSON tree (JsonHandler) with a typed, serde-derived writer
that produces the schema RFC 0015 defines: a flat harnesses[] array, checks{}/covers{}
buckets, failed_properties[] equal to checks.failure, warnings[], resolved_solver /
resolved_unwind, tools{}, configuration{}, harness_selection{}, run_state, and a
summary computed over COMPLETED harnesses only.

This breaks the old output format. The flag stays unstable.

Observable changes besides the new shape:

  • --export-json is gated by -Z export-json. -Z unstable-options alone is now an error.
  • An existing directory as the --export-json target is rejected during argument validation, as
    for --sarif. Both flags now share one check.
  • A missing parent directory of the target is created (as for --sarif).
  • The document is written to a temporary file in the target's directory, synced, and renamed onto
    the target. A failed write leaves any earlier file untouched and removes the temporary file.
    On Unix, an existing target keeps its mode and a new one gets 0o666 minus the umask, as with
    std::fs::write. If the target is a symbolic link, the link itself is replaced by the new file.
  • A run that stops before the end of verification leaves any earlier file in place, as before.
    This PR does not write RFC 0015's INCOMPLETE marker; that is tracked in Write the RFC 0015 INCOMPLETE marker before verification #4983. Until then,
    compare started_at to tell a stale file from the current run.
  • --exact now checks every --harness filter. Before, cargo kani --workspace --exact --harness common --harness typo ran when common existed in two crates, because the check
    compared match counts with filter counts. It now reports typo, with or without
    --export-json. See check_exact_filters_matched in metadata.rs.
  • build.rs records the git commit, a dirty flag and the rustc release for the document.
  • CBMC WARNING messages are kept (at most 20 per harness, 4096 characters each) only when
    --export-json is set. The cbmc --version probe is now a single function in version.rs.
  • docs/src/verification-results.md gets a short "JSON output" section.
  • Follows Fail verification when CBMC exits abnormally after reporting results #4910's rule for trusting CBMC's result array: a harness whose array is not trusted is
    exported as OUT_OF_MEMORY (status 137, or CBMC's own Out of memory report) or as CRASHED
    with CBMC's status as code, never as COMPLETED. The RFC's OUT_OF_MEMORY sentence is
    corrected to say so.

Fields of the old document that the new one does not have:

  • metadata.build_mode; metadata.version and metadata.timestamp are now schema_version and
    started_at.
  • project.workspace_root and project.output_dir (open questions in the RFC);
    project.crate_name is now crate_name on each harness.
  • tools.goto_cc, tools.goto_instrument, tools.goto_synthesizer and tools.solvers[].
    tools.rustc is now the toolchain release, e.g. 1.98.0-nightly (14210df0e 2026-05-31),
    taken at build time, not the output of kani-compiler --version.
  • Per harness: mangled_name, goto_file, source.end_line, is_ctor_based.
  • cbmc[].cbmc_metadata.os_info, cbmc[].configuration.object_bits and cbmc[].cbmc_stats
    (raw CBMC statistics; the RFC leaves them out). cbmc[].configuration.solver is now
    resolved_solver.
  • error_details[] and property_details[] (counts, including covered/uncovered for
    --coverage properties). outcome, failure_kind, checks and covers replace them.
  • The per-check column and the descriptions and locations of passing checks. Failed and
    unsupported-construct properties keep id, description, class, file and line.
  • verification_results.summary.status (completed / completed_with_fail_fast) and
    duration_ms; see run_state and wall_time_s.
  • coverage.enabled is now configuration.coverage_enabled.

Motivation

--export-json (#4472) predates RFC 0015 and does not match the schema the RFC specifies.
Consumers who want a stable, documented machine-readable result need the RFC's shape.

Known limitations

  • No INCOMPLETE marker yet (Write the RFC 0015 INCOMPLETE marker before verification #4983). A killed run can leave an earlier COMPLETE file.

  • kani_commit_dirty is true when tracked files differ from HEAD, staged or not. Untracked
    files are ignored, like the -dirty of kani --version --verbose. build.rs reruns only when
    .git/HEAD or .git/index changes, so an unstaged edit to a tracked file after the last build
    can leave the flag stale until the next rebuild.

  • resolved_solver and resolved_unwind describe what CBMC runs. They are computed from Kani's own
    flags followed by --cbmc-args, using CBMC's rules:

    • A named SMT flag is chosen by a fixed priority (--bitwuzla, --boolector, --cprover-smt2,
      --mathsat, --cvc5, --yices, --z3), whatever the argument order.
    • The first occurrence wins for a repeated --sat-solver, --external-sat-solver,
      --external-smt2-solver and --unwind. Kani passes --sat-solver cadical first, so
      --cbmc-args --sat-solver X changes nothing unless the run uses --solver minisat. In the same
      way a raw --cbmc-args --unwind N only applies when Kani passes no bound.
    • --external-sat-solver outranks --sat-solver. An SMT flag outranks both. CBMC rejects
      --external-sat-solver together with an SMT flag, and --incremental-smt2-solver together with
      a named one. Such a harness does not complete, and the exported value is a best guess.
    • When --external-smt2-solver <path> accompanies a named SMT flag, CBMC runs <path> with that
      solver's conventions. resolved_solver is then the path, which the RFC allows ("an arbitrary
      binary-path string"). The same holds for --incremental-smt2-solver <path>.
    • Bare --smt2 leaves the choice to CBMC and gives null.

    This PR corrects the solver and unwind text of RFC 0015 (rfc/src/rfcs/0015-export-json.md) to what
    CBMC runs. The table under Testing is the evidence.

Testing

Checked with CBMC 6.11.0 ($W1, $W2 are scripts that log their name and exit 1; hard.c has a
nondeterministic multiplication so that a solver must run):

command result
cbmc tiny.c --cvc5 --z3 --outfile o.txt, and --z3 --cvc5 both: ; Generated for CVC 5
cbmc tiny.c --sat-solver cadical --sat-solver notreal / reversed succeeds / unknown solver 'notreal'
cbmc hard.c --external-sat-solver $A --external-sat-solver $B / reversed A runs / B runs
cbmc loop5.c --unwind 3 --unwind 10 / reversed VERIFICATION FAILED / SUCCESSFUL
cbmc hard.c --z3 --external-smt2-solver $W1 and --external-smt2-solver $W1 --z3 W1 runs
cbmc hard.c --z3 --external-smt2-solver $W1 --external-smt2-solver $W2 / reversed W1 runs / W2 runs
cbmc hard.c --external-smt2-solver $W1 (no named SMT flag) required filename not provided; nothing runs
cbmc hard.c --incremental-smt2-solver $W1 W1 runs
cbmc hard.c --z3 --incremental-smt2-solver $W1, --external-sat-solver $W2 --z3 conflicting solver options
kani -Z export-json -Z unstable-options hard.rs --export-json o.json --solver z3 --cbmc-args --external-smt2-solver $W1 W1 runs; resolved_solver is its path

The Rust unit tests cover the writer's field mapping, the bucket partition, presence rules for each
outcome kind, the exported outcome for each CBMC exit status, the atomic write (including a value whose Serialize fails after partial output, and
the file mode), argument validation, and the solver/unwind logic above. The warning-capture test
runs a real child process through the streaming parser and cancels the parse like
--harness-timeout does. The run_cbmc_piped timeout wiring itself is exercised by the end-to-end
tests in a follow-up PR.

tests/json-handler runs the real tool (7 tests, including a two-crate workspace with
--exact --harness common --harness typo with and without --export-json, and a check that
tools.rustc and tools.cbmc have the documented shape). scripts/validate_json_export.py now
checks scalar types, array element shapes and the RFC's presence-by-outcome rules; fixtures for
valid non-COMPLETED exports, for malformed exports and for exports with unknown fields are in
tests/json-handler/schema-validation.
script-based-pre/harness_filter_no_match covers the zero-match error.

cargo test -p kani-driver, cargo clippy --all-targets -- -D warnings, cargo fmt --check and
scripts/run-autopep8.sh pass.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Replace the ad-hoc --export-json JSON tree (JsonHandler) with a typed,
serde-derived writer that produces the schema RFC 0015 defines: a flat
harnesses[] array, checks{}/covers{} buckets, failed_properties[] equal
to checks.failure, warnings[], resolved_solver/resolved_unwind, tools{},
configuration{}, harness_selection{}, run_state, and a summary computed
over COMPLETED harnesses only. The old output format is gone; the flag
stays unstable. Part of model-checking#942.

Other observable changes:

- --export-json is gated by -Z export-json instead of -Z unstable-options.
- An existing directory as the target is rejected during argument
  validation, with the same check as --sarif.
- A missing parent directory is created. The document is written to a
  temporary file next to the target, synced and renamed onto it; a failed
  write leaves an earlier file untouched. The file mode stays what
  std::fs::write gave (0o666 minus the umask).
- --exact now checks every --harness filter, with or without
  --export-json. A filter that matches in two crates no longer hides a
  sibling filter that matches nothing.
- build.rs records the git commit, a dirty flag (tracked files only) and
  the rustc release. tools.rustc is that release, not the output of
  kani-compiler --version.
- resolved_solver and resolved_unwind follow what CBMC runs for Kani's
  flags followed by --cbmc-args: fixed priority among named SMT solvers,
  first occurrence for repeated --sat-solver, --external-sat-solver,
  --external-smt2-solver and --unwind, and the executable path when
  --external-smt2-solver accompanies a named SMT flag.
  rfc/src/rfcs/0015-export-json.md is corrected to say so: its text had
  the last --cbmc-args override win, which CBMC does not do.
- CBMC WARNING messages are kept (capped) only with --export-json. There
  is one cbmc --version probe.
- docs/src/verification-results.md documents the flag.
- A harness whose CBMC result array is not trusted (the rule of model-checking#4910:
  exit status 0, 10, or 6 with an ERROR property) is exported as
  OUT_OF_MEMORY (status 137 or CBMC's own "Out of memory" report) or as
  CRASHED with CBMC's status as code, never as COMPLETED.
  rfc/src/rfcs/0015-export-json.md says so.

Dropped from the old document: metadata.build_mode, project.workspace_root
and output_dir, tools.goto_cc/goto_instrument/goto_synthesizer/solvers,
per-harness mangled_name, goto_file, end_line and is_ctor_based,
cbmc[].os_info, object_bits and cbmc_stats, error_details and
property_details, per-check columns and passing-check locations, and the
completed/completed_with_fail_fast status.

Tests: unit tests in kani-driver; the tests/json-handler suite (7 tests,
including a two-crate --exact regression with and without --export-json
and a check of the tools.* values); fixtures for valid non-COMPLETED
exports, malformed exports and unknown fields for
scripts/validate_json_export.py, which now checks scalar types, array
element shapes and the presence-by-outcome rules;
script-based-pre/harness_filter_no_match.
@ivmat
ivmat requested review from a team as code owners October 1, 2026 20:40
@ivmat

ivmat commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

i would also like to share the acceptance file for the RFC https://github.com/ivmat/kani/tree/export-json-rfc0015-acceptance/acceptance/rfc0015 based on https://github.com/ivmat/acceptance-format that i've been working on. i dont consider you would want it included in Kani repo - its not part of any PR and i dont request anything but i may be interesting to you.

in short, what it is - you can consider it a 'semantic coverage of the RFC claims' or in another way a 'glue format for the "artifact" and its specification, verification and more, ie json output implementation and its RFC and the verification of it. ive stared developing it for kani proofs in my DER repo and some tries on std lib challanges - early revision only covered verification but there are now more 'profiles' besides verification - we use 2 in this case.

Verification profile: "does the code do what each claim says?" Each claim carries tests and a control seen red, with a band. This is where all the evidence lives.
Conformance profile: "is each clause of the RFC covered?" The ledger holds no evidence of its own. Each covered clause borrows a verification claim's evidence by reference.

the format only gives weight to a statement if the proof or test can be shown to fail with levels in verification profile etc. i am expanding it in all directions (to be used as contract between software consumer and producer - but really first stable version will be minimal and simple for cases where you just use it to glue coverage).
long term idea is to mechanize work around it as much as possible and reduce semantically open parts (i.e just have a person or llm review minimal important, semantically open part; keep most of it mechanical so on rebuilds you get info for example which other parts of the repo may be impacted if you updated a specification, part of code or a test. its also really really useful in tracking whats been done, verified, and implemented and how. i'll continue developing it - kind of long term hope is to have chained acceptance files produced by all upstream libs and chained - and if a lib implements something with a strict specification even could come formally verified - and each claim could be accessed by downstream users...

The rest of the stack (ready on my fork; each link shows only that PR's own commit):

How accepance format was used

  1. Clause inventory. The RFC text is split into its normative units: 259 clauses, each with a stable id.
    Before any evidence was read, each clause was marked applicable, not applicable, or excluded, with a reason
    (210 applicable).
  2. Contract. The clauses are grouped into 15 requirements, each stating what evidence it needs (for
    example: a test plus a control that is seen to fail).
  3. Claims and controls. Every claim cites the test that backs it and a control: a small patch that breaks
    the production behaviour the claim is about. The control must turn the test red, and the red run is recorded.
    134 controls, all seen red at the final commit.
  4. Clause ledger. A generated manifest maps each clause to the tests that assert it. A clause counts as
    covered only when the tests assert every obligation in it, including exact values and "only/never" parts.
    111 of 210 applicable clauses are covered; the other 99 stay visible as gaps with a reason (for example the
    INCOMPLETE marker, which is not in this stack).
  5. Mechanical checks. Validators check the package: the hash of every evidence record, claim consistency,
    coverage per requirement. Coverage is reported as 12/15 requirements together with acceptable=False,
    because some requirements are only partly delivered.

What it caught:

  • The RFC's solver/unwind precedence sentence was wrong for repeated or mixed flags: CBMC picks named SMT
    solvers by a fixed priority and takes the first occurrence of a repeated flag. This PR exports what CBMC
    actually runs and corrects the RFC text.
  • An interaction with Fail verification when CBMC exits abnormally after reporting results #4910 (truncated CBMC results) was found by checking the stack against current main
    before opening. The export now follows Fail verification when CBMC exits abnormally after reporting results #4910's trusted-results rule, and the RFC's OUT_OF_MEMORY sentence is
    corrected to match.
  • Claims in my own PR descriptions that the code did not support (a removed "crashed" shape that never
    existed, a dropped field that was never exported, a bug fix that only existed in an unpublished draft). Each
    claim had to cite evidence, so they were found and removed.
  • Tests that could not fail. Several assertions passed even with the guarded code removed. Each now has a
    recorded red run, or the claim was lowered.
  • Over-counting. An early version of the clause ledger credited tests for more than they assert. The rule
    above (every obligation, exact values, "only/never") brought the covered count down to the honest number.

@github-actions github-actions Bot added the Z-EndToEndBenchCI Tag a PR to run benchmark CI label Oct 1, 2026
@feliperodri feliperodri self-assigned this Oct 2, 2026
@feliperodri
feliperodri requested a balanced review from Copilot October 2, 2026 01:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The export can leave stale completed results after interruption and can broaden existing output-file permissions.

Review effort: Balanced
Findings: 2 High severity

Open (2)
What changed in this PR

Migrates --export-json to RFC 0015’s typed schema and strengthens export reliability, provenance, validation, and harness filtering.

Changes:

  • Replaces the ad-hoc JSON handler with typed RFC-shaped serialization.
  • Adds atomic output, warning capture, solver/unwind provenance, and dedicated unstable gating.
  • Expands unit, integration, fixture, and schema-validation coverage.
File Description
kani-driver/​build.rs Records build provenance.
kani-driver/​src/​args/​mod.rs Validates export arguments and gate.
kani-driver/​src/​call_cbmc.rs Captures warnings and resolves CBMC outcomes.
kani-driver/​src/​cbmc_output_parser.rs Reconstructs CBMC property IDs.
kani-driver/​src/​cbmc_property_renderer.rs Removes obsolete JSON TODO.
kani-driver/​src/​frontend/​json_handler.rs Removes ad-hoc writer.
kani-driver/​src/​frontend/​mod.rs Retains typed schema module only.
kani-driver/​src/​frontend/​schema_utils.rs Implements RFC export and atomic writes.
kani-driver/​src/​frontend/​tests/​mod.rs Removes obsolete test module.
kani-driver/​src/​frontend/​tests/​schema_utils_test.rs Removes legacy schema tests.
kani-driver/​src/​harness_runner.rs Decouples verification from JSON mutation.
kani-driver/​src/​main.rs Writes typed export after verification.
kani-driver/​src/​metadata.rs Corrects exact-filter matching.
kani-driver/​src/​sarif.rs Shares path relativization.
kani-driver/​src/​version.rs Exposes toolchain and full CBMC versions.
kani_metadata/​src/​unstable.rs Adds the export-json feature gate.
scripts/​validate_json_export.py Validates RFC structure and outcomes.
docs/​src/​verification-results.md Documents JSON output usage.
rfc/​src/​rfcs/​0015-export-json.md Clarifies solver, unwind, and crash semantics.
tests/​script-based-pre/​harness_filter_no_match/​no_match.sh Verifies zero-match diagnostics.
tests/​json-handler/​basic-export/​test.sh Uses the dedicated gate.
tests/​json-handler/​cargo-export/​expected Updates expected output.
tests/​json-handler/​cargo-export/​test.sh Tests Cargo harness metadata.
tests/​json-handler/​fail-fast/​test.sh Tests partial fail-fast exports.
tests/​json-handler/​failed-verification/​test.sh Tests failed harness records.
tests/​json-handler/​multiple-harnesses/​test.sh Tests flat multi-harness output.
tests/​json-handler/​schema-validation/​check_fixtures.py Exercises validator failure cases.
tests/​json-handler/​schema-validation/​fixture-completed-failure.json Adds completed-failure fixture.
tests/​json-handler/​schema-validation/​fixture-no-harnesses-selected.json Adds empty-selection fixture.
tests/​json-handler/​schema-validation/​fixture-non-completed-outcomes.json Adds abnormal-outcome fixtures.
tests/​json-handler/​schema-validation/​kani_json_schema.json Defines the RFC template.
tests/​json-handler/​schema-validation/​test.sh Runs schema and fixture checks.
tests/​json-handler/​workspace-filter-diagnostic/​config.yml Registers workspace regression test.
tests/​json-handler/​workspace-filter-diagnostic/​test.sh Tests unmatched exact filters.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread kani-driver/src/main.rs Resolved
ivmat added 3 commits October 2, 2026 12:18
Renaming the temporary file onto the target replaced the target's mode
with 0o666 minus the umask, so a 0o600 export could become 0o644. Copy an
existing regular target's permissions onto the temporary file before the
rename, as `std::fs::write` (truncate in place) did. A new target still
gets 0o666 minus the umask.
Integrates upstream model-checking#4928 (only use CBMC results that are followed by
CBMC's overall status; new ExitStatus::IncompleteResults(i32)).

Conflicts:
- kani-driver/src/call_cbmc.rs: VerificationResult::from kept pub(crate)
  (used by the export) with upstream's new doc comment; upstream's
  prover-status rule, IncompleteResults, its message and its tests are
  kept unchanged.
- kani-driver/src/frontend/schema_utils.rs: upstream only touched the
  legacy export code this PR replaced; kept the RFC 0015 writer.

Export: ExitStatus::IncompleteResults(code) is exported as CRASHED with
code = the exit status and upstream's own message, like any other
untrusted result array. The ExitStatus match stays exhaustive. Unit
tests that build a CBMC output now append the prover status; a new
case set checks results without it are CRASHED for exit statuses 0, 10
and 6. The RFC sentence on which property arrays count now names the
prover-status condition.
After model-checking#4928, results CBMC did not follow with cProverStatus are not used,
whatever the exit status, so a CRASHED harness can carry code 0.
@ivmat

ivmat commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Merged main to resolve the conflict with #4928. Its new ExitStatus::IncompleteResults is exported as CRASHED with the exit status as code (From<&ExitStatus> in schema_utils.rs), so code can now be 0; the RFC says so (938a3ce). The unit test export_outcome_follows_cbmc_exit_status_and_results covers exit 0, 10 and 6 without cProverStatus.

@feliperodri feliperodri left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I ran --export-json across the cases RFC 0015 cares about: success, failure, contradictory assumes, should_panic both ways, the three cover states, reached and unreached unsupported constructs, the #4719 quantifier override, --harness-timeout, a CBMC usage error, a SIGKILLed CBMC and a CBMC that reports Out of memory, concrete playback, -j 3, autoharness, --coverage, --fail-fast, and a two-crate cargo kani --workspace with same-named harnesses. Every document passed validate_json_export.py. I also checked them against a small script for the RFC identities the validator doesn't cover (bucket sums, n_failed, failed_properties == checks.failure, the summary sums, run_state against matched_count, sort order). None failed. -j 3 output matches the serial run once the timestamps and timings are removed.

Before merge, I'd like a tracking issue for the INCOMPLETE marker and a sentence in verification-results.md saying this version doesn't write one. I can reproduce a stale COMPLETE file by killing the driver mid-run (details inline). It isn't a regression, but #4731 was closed with the marker as its fix, and the docs now point readers at an RFC that promises one.

The other inline comments are places where the RFC is silent or says something the code doesn't do: workspace-relative file under cargo kani, the mangled contracted_function_name, and whether wall_time_s includes the build. RFC text fixes would settle all three. The duplicate entry in enabled_unstable_features is a nit.

On compatibility: nothing in the repo reads the old document (I searched tools/, scripts/, .github/ and benchcomp). Old readers fail on the missing metadata key instead of misreading the new shape. -Z unstable-options --export-json now gets an error that names -Z export-json, which is enough of a migration note for an unstable flag. schema_version: "0.1.0" is the first version of this schema, so there's nothing to bump.

Not verified: an is_bounded: true harness, macOS, and the full regression suite (CI is green on all of them).

Comment thread docs/src/verification-results.md Resolved
Comment thread kani-driver/src/main.rs Resolved
Comment thread kani-driver/src/metadata.rs Resolved
@feliperodri feliperodri assigned ivmat and unassigned feliperodri Oct 5, 2026
ivmat added 2 commits October 6, 2026 21:41
Collect enabled_unstable_features into a BTreeSet, so
`-Z export-json -Z export-json` exports the flag once. The set is
already sorted, so the explicit sort goes. basic-export passes the flag
twice and checks the exported list.
Say in verification-results.md that this version never writes the
INCOMPLETE marker, so a killed run can leave a stale file. In RFC 0015,
state the base of harnesses[].file under cargo kani and kani, that
contracted_function_name is an internal mangled symbol, and what
started_at and wall_time_s cover.
ivmat and others added 3 commits October 6, 2026 22:40
AssignsContract::contracted_function_name holds the mangled symbol of
the generated closure that carries the contract, not the function the
contract is for. Say so, as the RFC 0015 field row now does.
@feliperodri
feliperodri enabled auto-merge October 21, 2026 22:14
@feliperodri
feliperodri added this pull request to the merge queue Oct 21, 2026
Merged via the queue into model-checking:main with commit 1f034d9 Oct 21, 2026
33 checks passed
calef added a commit to calef/kani that referenced this pull request Oct 22, 2026
The merge of main brought the RFC 0015 --export-json shape (model-checking#4941), which
reports `target` as the triple Kani was built for and gates the flag behind
`-Z export-json`. With --target that triple is the wrong one, and the
target_riscv64 script test, which checks that the export records the
verification target, no longer ran. Take `target` from
`verification_target()`, say so in the RFC's field table, and update the
script test for the new key and the new gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feliperodri added a commit to feliperodri/kani that referenced this pull request Nov 7, 2026
…--export-json

Kani 0.68.0 and older source builds gate --export-json behind -Z unstable-options
and reject -Z export-json; builds since the RFC 0015 export (model-checking#4941) require
-Z export-json and reject -Z unstable-options alone. Probe once with
`kani -Z export-json --version` (Kani validates -Z values before it acts on
--version), cache the answer, and use it in every script, in kani-mcp, in the plan's
AutoHarness command, and when replaying recorded commands. check-kani-version.py
prints export_flags. KANI_PLUGIN_EXPORT_FLAG overrides the probe.
feliperodri added a commit to feliperodri/kani that referenced this pull request Nov 7, 2026
Kani 0.68.0 and older builds write a tree (harness_metadata, verification_results,
property_details); Kani since model-checking#4941 writes a flat document (harnesses, configuration).
The plugin read only the flat one, so a passing run on the older Kani summarized as
0 harnesses and all_ok false.

One reader per language (scripts/kani_export.py, tools/kani-mcp/src/export.rs) converts the
tree to the flat shape and is used by summarize-results, check-vacuity, check-specs,
evidence, run-harnesses, the Crate Verifiability Map and every kani-mcp tool. What the tree
does not record is marked: the run configuration (summaries say weakened checks cannot be
detected from the export), and stubs, which are read from the harness attributes in the
source (stubs_source: source-scan, autoharness-generated, or unknown). The source is
accepted only if the harness fn is where the export says.

check-kani-version.py and kani_get_version now read the export of a trivial run and fail
when it cannot be understood. Tests hold one crate's export in both shapes and check that the
answers agree; a smoke test does the same against the Kani under test.
wodex1nhaoIeng pushed a commit to wodex1nhaoIeng/kani that referenced this pull request Nov 10, 2026
model-checking#5062)

Resolves model-checking#4983. Part of model-checking#942 (RFC 0015, model-checking#4727). Follows model-checking#4941.

## Description

RFC 0015 says an earlier `--export-json` file is replaced by an
`INCOMPLETE` marker when verification
begins. Without it, a run that is killed during verification leaves the
earlier `COMPLETE` document in
place, and that document looks current (see the
[model-checking#4941
review](model-checking#4941 (comment))).

- After the build and harness selection succeed, Kani builds the run
context once and writes the
marker with the same atomic write as the final document. After
verification, the final document
from the same context replaces it, so `started_at` and the other shared
fields are equal in both.
- The marker has the fields of the RFC presence matrix and `run_state:
"INCOMPLETE"`, and no
  `outcome`, `wall_time_s`, `summary` or `harnesses`.
- A failed marker write stops the run before any harness is verified. A
build error or a rejected
harness selection (a filter that matches nothing) leaves an earlier file
in place.
- `wall_time_s` now also covers the tool version probes and the marker
write. The build is still not
  included.

This implements the marker that the RFC proposes. The RFC's open
question "marker or delete-up-front"
is unchanged; the flag stays unstable. The docs and the RFC text are
updated to match.

Readers of the export should treat a `run_state` other than `COMPLETE`
as incomplete evidence. For example,
kani-mcp in model-checking#5059 deletes any earlier export before a run and reports a
timeout only when no export exists.
With this change, a run that stops after the marker write leaves an
`INCOMPLETE` export, so the existence of
the file alone does not show that verification finished.

## Testing

- Unit tests: the marker's exact key set, and null metadata serialized
as null; a marker and a final
  document from one context have equal shared fields.
- `tests/json-handler/incomplete-marker` runs the real driver with a
fake `cbmc` that waits for a
release file. Killing the driver leaves the marker; releasing it gives a
final document with the
marker's `started_at`; a bad export path and a rejected `--harness`
filter stop the run before
  CBMC verification starts.
- Each of these mutations makes a test fail: the marker written after
verification, the marker
written before harness selection, a marker write error ignored,
`summary: null` in the marker, and
  a recomputed `started_at`.
- Full regression, `cargo clippy` and `cargo fmt --check` pass at
1b6fefa.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 and MIT licenses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Z-EndToEndBenchCI Tag a PR to run benchmark CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants