Repository navigation
Migrate --export-json to the RFC 0015 document shape - #4941
Conversation
Replace the ad-hoc --export-json JSON tree (JsonHandler) with a typed,
serde-derived writer that produces the schema RFC 0015 defines: a flat
harnesses[] array, checks{}/covers{} buckets, failed_properties[] equal
to checks.failure, warnings[], resolved_solver/resolved_unwind, tools{},
configuration{}, harness_selection{}, run_state, and a summary computed
over COMPLETED harnesses only. The old output format is gone; the flag
stays unstable. Part of model-checking#942.
Other observable changes:
- --export-json is gated by -Z export-json instead of -Z unstable-options.
- An existing directory as the target is rejected during argument
validation, with the same check as --sarif.
- A missing parent directory is created. The document is written to a
temporary file next to the target, synced and renamed onto it; a failed
write leaves an earlier file untouched. The file mode stays what
std::fs::write gave (0o666 minus the umask).
- --exact now checks every --harness filter, with or without
--export-json. A filter that matches in two crates no longer hides a
sibling filter that matches nothing.
- build.rs records the git commit, a dirty flag (tracked files only) and
the rustc release. tools.rustc is that release, not the output of
kani-compiler --version.
- resolved_solver and resolved_unwind follow what CBMC runs for Kani's
flags followed by --cbmc-args: fixed priority among named SMT solvers,
first occurrence for repeated --sat-solver, --external-sat-solver,
--external-smt2-solver and --unwind, and the executable path when
--external-smt2-solver accompanies a named SMT flag.
rfc/src/rfcs/0015-export-json.md is corrected to say so: its text had
the last --cbmc-args override win, which CBMC does not do.
- CBMC WARNING messages are kept (capped) only with --export-json. There
is one cbmc --version probe.
- docs/src/verification-results.md documents the flag.
- A harness whose CBMC result array is not trusted (the rule of model-checking#4910:
exit status 0, 10, or 6 with an ERROR property) is exported as
OUT_OF_MEMORY (status 137 or CBMC's own "Out of memory" report) or as
CRASHED with CBMC's status as code, never as COMPLETED.
rfc/src/rfcs/0015-export-json.md says so.
Dropped from the old document: metadata.build_mode, project.workspace_root
and output_dir, tools.goto_cc/goto_instrument/goto_synthesizer/solvers,
per-harness mangled_name, goto_file, end_line and is_ctor_based,
cbmc[].os_info, object_bits and cbmc_stats, error_details and
property_details, per-check columns and passing-check locations, and the
completed/completed_with_fail_fast status.
Tests: unit tests in kani-driver; the tests/json-handler suite (7 tests,
including a two-crate --exact regression with and without --export-json
and a check of the tools.* values); fixtures for valid non-COMPLETED
exports, malformed exports and unknown fields for
scripts/validate_json_export.py, which now checks scalar types, array
element shapes and the presence-by-outcome rules;
script-based-pre/harness_filter_no_match.
|
i would also like to share the acceptance file for the RFC https://github.com/ivmat/kani/tree/export-json-rfc0015-acceptance/acceptance/rfc0015 based on https://github.com/ivmat/acceptance-format that i've been working on. i dont consider you would want it included in Kani repo - its not part of any PR and i dont request anything but i may be interesting to you. in short, what it is - you can consider it a 'semantic coverage of the RFC claims' or in another way a 'glue format for the "artifact" and its specification, verification and more, ie json output implementation and its RFC and the verification of it. ive stared developing it for kani proofs in my DER repo and some tries on std lib challanges - early revision only covered verification but there are now more 'profiles' besides verification - we use 2 in this case.
the format only gives weight to a statement if the proof or test can be shown to fail with levels in verification profile etc. i am expanding it in all directions (to be used as contract between software consumer and producer - but really first stable version will be minimal and simple for cases where you just use it to glue coverage). The rest of the stack (ready on my fork; each link shows only that PR's own commit):
How accepance format was used
What it caught:
|
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The export can leave stale completed results after interruption and can broaden existing output-file permissions.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Migrates --export-json to RFC 0015’s typed schema and strengthens export reliability, provenance, validation, and harness filtering.
Changes:
- Replaces the ad-hoc JSON handler with typed RFC-shaped serialization.
- Adds atomic output, warning capture, solver/unwind provenance, and dedicated unstable gating.
- Expands unit, integration, fixture, and schema-validation coverage.
| File | Description |
|---|---|
kani-driver/build.rs |
Records build provenance. |
kani-driver/src/args/mod.rs |
Validates export arguments and gate. |
kani-driver/src/call_cbmc.rs |
Captures warnings and resolves CBMC outcomes. |
kani-driver/src/cbmc_output_parser.rs |
Reconstructs CBMC property IDs. |
kani-driver/src/cbmc_property_renderer.rs |
Removes obsolete JSON TODO. |
kani-driver/src/frontend/json_handler.rs |
Removes ad-hoc writer. |
kani-driver/src/frontend/mod.rs |
Retains typed schema module only. |
kani-driver/src/frontend/schema_utils.rs |
Implements RFC export and atomic writes. |
kani-driver/src/frontend/tests/mod.rs |
Removes obsolete test module. |
kani-driver/src/frontend/tests/schema_utils_test.rs |
Removes legacy schema tests. |
kani-driver/src/harness_runner.rs |
Decouples verification from JSON mutation. |
kani-driver/src/main.rs |
Writes typed export after verification. |
kani-driver/src/metadata.rs |
Corrects exact-filter matching. |
kani-driver/src/sarif.rs |
Shares path relativization. |
kani-driver/src/version.rs |
Exposes toolchain and full CBMC versions. |
kani_metadata/src/unstable.rs |
Adds the export-json feature gate. |
scripts/validate_json_export.py |
Validates RFC structure and outcomes. |
docs/src/verification-results.md |
Documents JSON output usage. |
rfc/src/rfcs/0015-export-json.md |
Clarifies solver, unwind, and crash semantics. |
tests/script-based-pre/harness_filter_no_match/no_match.sh |
Verifies zero-match diagnostics. |
tests/json-handler/basic-export/test.sh |
Uses the dedicated gate. |
tests/json-handler/cargo-export/expected |
Updates expected output. |
tests/json-handler/cargo-export/test.sh |
Tests Cargo harness metadata. |
tests/json-handler/fail-fast/test.sh |
Tests partial fail-fast exports. |
tests/json-handler/failed-verification/test.sh |
Tests failed harness records. |
tests/json-handler/multiple-harnesses/test.sh |
Tests flat multi-harness output. |
tests/json-handler/schema-validation/check_fixtures.py |
Exercises validator failure cases. |
tests/json-handler/schema-validation/fixture-completed-failure.json |
Adds completed-failure fixture. |
tests/json-handler/schema-validation/fixture-no-harnesses-selected.json |
Adds empty-selection fixture. |
tests/json-handler/schema-validation/fixture-non-completed-outcomes.json |
Adds abnormal-outcome fixtures. |
tests/json-handler/schema-validation/kani_json_schema.json |
Defines the RFC template. |
tests/json-handler/schema-validation/test.sh |
Runs schema and fixture checks. |
tests/json-handler/workspace-filter-diagnostic/config.yml |
Registers workspace regression test. |
tests/json-handler/workspace-filter-diagnostic/test.sh |
Tests unmatched exact filters. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Renaming the temporary file onto the target replaced the target's mode with 0o666 minus the umask, so a 0o600 export could become 0o644. Copy an existing regular target's permissions onto the temporary file before the rename, as `std::fs::write` (truncate in place) did. A new target still gets 0o666 minus the umask.
Integrates upstream model-checking#4928 (only use CBMC results that are followed by CBMC's overall status; new ExitStatus::IncompleteResults(i32)). Conflicts: - kani-driver/src/call_cbmc.rs: VerificationResult::from kept pub(crate) (used by the export) with upstream's new doc comment; upstream's prover-status rule, IncompleteResults, its message and its tests are kept unchanged. - kani-driver/src/frontend/schema_utils.rs: upstream only touched the legacy export code this PR replaced; kept the RFC 0015 writer. Export: ExitStatus::IncompleteResults(code) is exported as CRASHED with code = the exit status and upstream's own message, like any other untrusted result array. The ExitStatus match stays exhaustive. Unit tests that build a CBMC output now append the prover status; a new case set checks results without it are CRASHED for exit statuses 0, 10 and 6. The RFC sentence on which property arrays count now names the prover-status condition.
After model-checking#4928, results CBMC did not follow with cProverStatus are not used, whatever the exit status, so a CRASHED harness can carry code 0.
|
Merged main to resolve the conflict with #4928. Its new ExitStatus::IncompleteResults is exported as CRASHED with the exit status as code (From<&ExitStatus> in schema_utils.rs), so code can now be 0; the RFC says so (938a3ce). The unit test export_outcome_follows_cbmc_exit_status_and_results covers exit 0, 10 and 6 without cProverStatus. |
feliperodri
left a comment
There was a problem hiding this comment.
Approving. I ran --export-json across the cases RFC 0015 cares about: success, failure, contradictory assumes, should_panic both ways, the three cover states, reached and unreached unsupported constructs, the #4719 quantifier override, --harness-timeout, a CBMC usage error, a SIGKILLed CBMC and a CBMC that reports Out of memory, concrete playback, -j 3, autoharness, --coverage, --fail-fast, and a two-crate cargo kani --workspace with same-named harnesses. Every document passed validate_json_export.py. I also checked them against a small script for the RFC identities the validator doesn't cover (bucket sums, n_failed, failed_properties == checks.failure, the summary sums, run_state against matched_count, sort order). None failed. -j 3 output matches the serial run once the timestamps and timings are removed.
Before merge, I'd like a tracking issue for the INCOMPLETE marker and a sentence in verification-results.md saying this version doesn't write one. I can reproduce a stale COMPLETE file by killing the driver mid-run (details inline). It isn't a regression, but #4731 was closed with the marker as its fix, and the docs now point readers at an RFC that promises one.
The other inline comments are places where the RFC is silent or says something the code doesn't do: workspace-relative file under cargo kani, the mangled contracted_function_name, and whether wall_time_s includes the build. RFC text fixes would settle all three. The duplicate entry in enabled_unstable_features is a nit.
On compatibility: nothing in the repo reads the old document (I searched tools/, scripts/, .github/ and benchcomp). Old readers fail on the missing metadata key instead of misreading the new shape. -Z unstable-options --export-json now gets an error that names -Z export-json, which is enough of a migration note for an unstable flag. schema_version: "0.1.0" is the first version of this schema, so there's nothing to bump.
Not verified: an is_bounded: true harness, macOS, and the full regression suite (CI is green on all of them).
Collect enabled_unstable_features into a BTreeSet, so `-Z export-json -Z export-json` exports the flag once. The set is already sorted, so the explicit sort goes. basic-export passes the flag twice and checks the exported list.
Say in verification-results.md that this version never writes the INCOMPLETE marker, so a killed run can leave a stale file. In RFC 0015, state the base of harnesses[].file under cargo kani and kani, that contracted_function_name is an internal mangled symbol, and what started_at and wall_time_s cover.
AssignsContract::contracted_function_name holds the mangled symbol of the generated closure that carries the contract, not the function the contract is for. Say so, as the RFC 0015 field row now does.
The merge of main brought the RFC 0015 --export-json shape (model-checking#4941), which reports `target` as the triple Kani was built for and gates the flag behind `-Z export-json`. With --target that triple is the wrong one, and the target_riscv64 script test, which checks that the export records the verification target, no longer ran. Take `target` from `verification_target()`, say so in the RFC's field table, and update the script test for the new key and the new gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…--export-json Kani 0.68.0 and older source builds gate --export-json behind -Z unstable-options and reject -Z export-json; builds since the RFC 0015 export (model-checking#4941) require -Z export-json and reject -Z unstable-options alone. Probe once with `kani -Z export-json --version` (Kani validates -Z values before it acts on --version), cache the answer, and use it in every script, in kani-mcp, in the plan's AutoHarness command, and when replaying recorded commands. check-kani-version.py prints export_flags. KANI_PLUGIN_EXPORT_FLAG overrides the probe.
Kani 0.68.0 and older builds write a tree (harness_metadata, verification_results, property_details); Kani since model-checking#4941 writes a flat document (harnesses, configuration). The plugin read only the flat one, so a passing run on the older Kani summarized as 0 harnesses and all_ok false. One reader per language (scripts/kani_export.py, tools/kani-mcp/src/export.rs) converts the tree to the flat shape and is used by summarize-results, check-vacuity, check-specs, evidence, run-harnesses, the Crate Verifiability Map and every kani-mcp tool. What the tree does not record is marked: the run configuration (summaries say weakened checks cannot be detected from the export), and stubs, which are read from the harness attributes in the source (stubs_source: source-scan, autoharness-generated, or unknown). The source is accepted only if the harness fn is where the export says. check-kani-version.py and kani_get_version now read the export of a trivial run and fail when it cannot be understood. Tests hold one crate's export in both shapes and check that the answers agree; a smoke test does the same against the Kani under test.
model-checking#5062) Resolves model-checking#4983. Part of model-checking#942 (RFC 0015, model-checking#4727). Follows model-checking#4941. ## Description RFC 0015 says an earlier `--export-json` file is replaced by an `INCOMPLETE` marker when verification begins. Without it, a run that is killed during verification leaves the earlier `COMPLETE` document in place, and that document looks current (see the [model-checking#4941 review](model-checking#4941 (comment))). - After the build and harness selection succeed, Kani builds the run context once and writes the marker with the same atomic write as the final document. After verification, the final document from the same context replaces it, so `started_at` and the other shared fields are equal in both. - The marker has the fields of the RFC presence matrix and `run_state: "INCOMPLETE"`, and no `outcome`, `wall_time_s`, `summary` or `harnesses`. - A failed marker write stops the run before any harness is verified. A build error or a rejected harness selection (a filter that matches nothing) leaves an earlier file in place. - `wall_time_s` now also covers the tool version probes and the marker write. The build is still not included. This implements the marker that the RFC proposes. The RFC's open question "marker or delete-up-front" is unchanged; the flag stays unstable. The docs and the RFC text are updated to match. Readers of the export should treat a `run_state` other than `COMPLETE` as incomplete evidence. For example, kani-mcp in model-checking#5059 deletes any earlier export before a run and reports a timeout only when no export exists. With this change, a run that stops after the marker write leaves an `INCOMPLETE` export, so the existence of the file alone does not show that verification finished. ## Testing - Unit tests: the marker's exact key set, and null metadata serialized as null; a marker and a final document from one context have equal shared fields. - `tests/json-handler/incomplete-marker` runs the real driver with a fake `cbmc` that waits for a release file. Killing the driver leaves the marker; releasing it gives a final document with the marker's `started_at`; a bad export path and a rejected `--harness` filter stop the run before CBMC verification starts. - Each of these mutations makes a test fail: the marker written after verification, the marker written before harness selection, a marker write error ignored, `summary: null` in the marker, and a recomputed `started_at`. - Full regression, `cargo clippy` and `cargo fmt --check` pass at 1b6fefa. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

Part of #942 (RFC 0015, #4727).
Description
Replaces the ad-hoc
--export-jsonJSON tree (JsonHandler) with a typed, serde-derived writerthat produces the schema RFC 0015 defines: a flat
harnesses[]array,checks{}/covers{}buckets,
failed_properties[]equal tochecks.failure,warnings[],resolved_solver/resolved_unwind,tools{},configuration{},harness_selection{},run_state, and asummarycomputed overCOMPLETEDharnesses only.This breaks the old output format. The flag stays unstable.
Observable changes besides the new shape:
--export-jsonis gated by-Z export-json.-Z unstable-optionsalone is now an error.--export-jsontarget is rejected during argument validation, asfor
--sarif. Both flags now share one check.--sarif).the target. A failed write leaves any earlier file untouched and removes the temporary file.
On Unix, an existing target keeps its mode and a new one gets 0o666 minus the umask, as with
std::fs::write. If the target is a symbolic link, the link itself is replaced by the new file.This PR does not write RFC 0015's
INCOMPLETEmarker; that is tracked in Write the RFC 0015 INCOMPLETE marker before verification #4983. Until then,compare
started_atto tell a stale file from the current run.--exactnow checks every--harnessfilter. Before,cargo kani --workspace --exact --harness common --harness typoran whencommonexisted in two crates, because the checkcompared match counts with filter counts. It now reports
typo, with or without--export-json. Seecheck_exact_filters_matchedinmetadata.rs.build.rsrecords the git commit, a dirty flag and the rustc release for the document.WARNINGmessages are kept (at most 20 per harness, 4096 characters each) only when--export-jsonis set. Thecbmc --versionprobe is now a single function inversion.rs.docs/src/verification-results.mdgets a short "JSON output" section.exported as
OUT_OF_MEMORY(status 137, or CBMC's ownOut of memoryreport) or asCRASHEDwith CBMC's status as
code, never asCOMPLETED. The RFC'sOUT_OF_MEMORYsentence iscorrected to say so.
Fields of the old document that the new one does not have:
metadata.build_mode;metadata.versionandmetadata.timestampare nowschema_versionandstarted_at.project.workspace_rootandproject.output_dir(open questions in the RFC);project.crate_nameis nowcrate_nameon each harness.tools.goto_cc,tools.goto_instrument,tools.goto_synthesizerandtools.solvers[].tools.rustcis now the toolchain release, e.g.1.98.0-nightly (14210df0e 2026-05-31),taken at build time, not the output of
kani-compiler --version.mangled_name,goto_file,source.end_line,is_ctor_based.cbmc[].cbmc_metadata.os_info,cbmc[].configuration.object_bitsandcbmc[].cbmc_stats(raw CBMC statistics; the RFC leaves them out).
cbmc[].configuration.solveris nowresolved_solver.error_details[]andproperty_details[](counts, includingcovered/uncoveredfor--coverageproperties).outcome,failure_kind,checksandcoversreplace them.columnand the descriptions and locations of passing checks. Failed andunsupported-construct properties keep
id,description,class,fileandline.verification_results.summary.status(completed/completed_with_fail_fast) andduration_ms; seerun_stateandwall_time_s.coverage.enabledis nowconfiguration.coverage_enabled.Motivation
--export-json(#4472) predates RFC 0015 and does not match the schema the RFC specifies.Consumers who want a stable, documented machine-readable result need the RFC's shape.
Known limitations
No
INCOMPLETEmarker yet (Write the RFC 0015 INCOMPLETE marker before verification #4983). A killed run can leave an earlierCOMPLETEfile.kani_commit_dirtyis true when tracked files differ fromHEAD, staged or not. Untrackedfiles are ignored, like the
-dirtyofkani --version --verbose.build.rsreruns only when.git/HEADor.git/indexchanges, so an unstaged edit to a tracked file after the last buildcan leave the flag stale until the next rebuild.
resolved_solverandresolved_unwinddescribe what CBMC runs. They are computed from Kani's ownflags followed by
--cbmc-args, using CBMC's rules:--bitwuzla,--boolector,--cprover-smt2,--mathsat,--cvc5,--yices,--z3), whatever the argument order.--sat-solver,--external-sat-solver,--external-smt2-solverand--unwind. Kani passes--sat-solver cadicalfirst, so--cbmc-args --sat-solver Xchanges nothing unless the run uses--solver minisat. In the sameway a raw
--cbmc-args --unwind Nonly applies when Kani passes no bound.--external-sat-solveroutranks--sat-solver. An SMT flag outranks both. CBMC rejects--external-sat-solvertogether with an SMT flag, and--incremental-smt2-solvertogether witha named one. Such a harness does not complete, and the exported value is a best guess.
--external-smt2-solver <path>accompanies a named SMT flag, CBMC runs<path>with thatsolver's conventions.
resolved_solveris then the path, which the RFC allows ("an arbitrarybinary-path string"). The same holds for
--incremental-smt2-solver <path>.--smt2leaves the choice to CBMC and givesnull.This PR corrects the solver and unwind text of RFC 0015 (
rfc/src/rfcs/0015-export-json.md) to whatCBMC runs. The table under Testing is the evidence.
Testing
Checked with CBMC 6.11.0 (
$W1,$W2are scripts that log their name and exit 1;hard.chas anondeterministic multiplication so that a solver must run):
cbmc tiny.c --cvc5 --z3 --outfile o.txt, and--z3 --cvc5; Generated for CVC 5cbmc tiny.c --sat-solver cadical --sat-solver notreal/ reversedunknown solver 'notreal'cbmc hard.c --external-sat-solver $A --external-sat-solver $B/ reversedAruns /Brunscbmc loop5.c --unwind 3 --unwind 10/ reversedVERIFICATION FAILED/SUCCESSFULcbmc hard.c --z3 --external-smt2-solver $W1and--external-smt2-solver $W1 --z3W1runscbmc hard.c --z3 --external-smt2-solver $W1 --external-smt2-solver $W2/ reversedW1runs /W2runscbmc hard.c --external-smt2-solver $W1(no named SMT flag)required filename not provided; nothing runscbmc hard.c --incremental-smt2-solver $W1W1runscbmc hard.c --z3 --incremental-smt2-solver $W1,--external-sat-solver $W2 --z3conflicting solver optionskani -Z export-json -Z unstable-options hard.rs --export-json o.json --solver z3 --cbmc-args --external-smt2-solver $W1W1runs;resolved_solveris its pathThe Rust unit tests cover the writer's field mapping, the bucket partition, presence rules for each
outcome kind, the exported outcome for each CBMC exit status, the atomic write (including a value whose
Serializefails after partial output, andthe file mode), argument validation, and the solver/unwind logic above. The warning-capture test
runs a real child process through the streaming parser and cancels the parse like
--harness-timeoutdoes. Therun_cbmc_pipedtimeout wiring itself is exercised by the end-to-endtests in a follow-up PR.
tests/json-handlerruns the real tool (7 tests, including a two-crate workspace with--exact --harness common --harness typowith and without--export-json, and a check thattools.rustcandtools.cbmchave the documented shape).scripts/validate_json_export.pynowchecks scalar types, array element shapes and the RFC's presence-by-outcome rules; fixtures for
valid non-
COMPLETEDexports, for malformed exports and for exports with unknown fields are intests/json-handler/schema-validation.script-based-pre/harness_filter_no_matchcovers the zero-match error.cargo test -p kani-driver,cargo clippy --all-targets -- -D warnings,cargo fmt --checkandscripts/run-autopep8.shpass.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.