Repository navigation
Allow user to specify CBMC's solver - #2088
Conversation
84adc14 to
97b40f8
Compare
|
I would prefer if the attribute is based on an ID, not a path. I don't think it's a good practice to hardcode paths in the code, since it is hard to ensure that the path works for every developer and the CI. To make this extensible for users that want to try something new that doesn't ship with Kani, I would prefer adding installation instructions, command line or a utility that configures a new engine via its path into a format / path that Kani expects. |
Point taken. A For the purposes of allowing the user to control CBMC's SAT solver specifically (which this PR is targeting), I can think of two options:
The advantage of option 1 is that it would keep the more valuable/more generic The advantage of option 2 is that the user only needs to learn about one attribute for specifying the solver (i.e. fewer control knobs) which makes the usage simpler. Thoughts? |
|
Option 2 sounds better. |
|
Just a nit... maybe use "::" as a convention, it is also less likely that "::" will be in the same of a binary. I was thinking, since we only have cbmc today, we could consider that things that an id that doesn't have "::" uses cbmc. I.e.: Another option would be to start with what you have, and once we add more options, we use key / value for the attribute. And we support the previous mode as legacy purpose. I.e.: |
|
Thanks for pushing on this, @zhassan-aws ! Why are we adding a new attribute? We could just make This idea comes from @fzaiser in #1656, and in my opinion is much cleaner than the other options considered here. It'd work in a similar way to @celinval 's last suggestion, assuming I don't think engines and solvers should be coupled in any of these options neither (i.e., things like |
I fully agree with making attributes arguments to |
97b40f8 to
1d3738c
Compare
|
Are we going in the direction of adding a new attribute vs attribute arguments? |
We agreed that we will support both approaches. |
|
I have a few comments regarding the UX from the PR description:
|
Make sense.
Good idea.
TBH, I don't like |
Sounds good to me. |
|
Added minisat in e426f79. For Is this sufficient? |
I updated the PR to not use strings, e.g. |
Done. This only works for the attribute though, but not the command-line option :( It seems that to support passing |
Thanks, @zhassan-aws ! One question: How would the user specify the solver in |
I'm not sure :) We could perhaps do |
celinval
left a comment
There was a problem hiding this comment.
It looks good to me. Just some minor tweaks of the error messages.
| CbmcSolver::Custom(custom_solver) => { | ||
| // Check if the specified binary exists in path | ||
| if which::which(custom_solver).is_err() { | ||
| eprintln!( |
There was a problem hiding this comment.
I was wondering if we should merge this eprintln!() and the bail() statements. If you do want to keep them separate, can you please use util::error instead to print this message so we keep our formatting consistent?
I think we are using stdout there but you can change it if you prefer.
There was a problem hiding this comment.
Removed the eprintln and moved its message to bail.
| Some(CbmcSolver::Custom(lit.token_lit.symbol.to_string())) | ||
| } | ||
| _ => { | ||
| invalid_arg_err(attr); |
There was a problem hiding this comment.
Can you make this message more specific to the custom option?
There was a problem hiding this comment.
This message is not specific to custom. It is emitted when the argument received is neither a MetaItemKind::Word or MetaItemKind::NameValue.
| eprintln!( | ||
| "Error: The specified solver \"{custom_solver}\" was not found in path" | ||
| ); | ||
| bail!("cbmc solver argument handling failed") |
There was a problem hiding this comment.
If you want to keep this separate from the error print, can you please make this more user friendly. Something like, "Custom solver resolution failed".
| error: The `#[kani::solver]` attribute expects a single argument. Got 0 arguments. | ||
| test.rs:5:1 | ||
| | | ||
| 5 | #[kani::solver] |
There was a problem hiding this comment.
nit: remove the line number from the expected files and add "" to the end of each line.
adpaco
left a comment
There was a problem hiding this comment.
Just a few comments about error messages.
I'd like us to iterate on the best option to pass the path for custom solvers. In my opinion, there shouldn't be any differences between the CLI and the annotation, which AFAIK is not true at the moment. But I don't want this to block the contribution since it's a special, non-required case for me.
| Deserialize | ||
| )] | ||
| #[strum(serialize_all = "snake_case")] | ||
| pub enum CbmcSolver { |
There was a problem hiding this comment.
What's the reason for this to be CbmcSolver and not just Solver?
There was a problem hiding this comment.
Since the solver is currently specific for CBMC, and we're not yet clear on future plans, and whether this solver will apply to other engines as well, I added CBMC to its name.
There was a problem hiding this comment.
I tend to agree with @adpaco-aws here, but this is an implementation detail that is internal to our code, so we can easily rename this later.
I implemented a clap parser for the @celinval this is something that you have requested as well. |
adpaco
left a comment
There was a problem hiding this comment.
Thanks, @zhassan-aws ! Looks great overall, just a few suggestions 😄
Co-authored-by: Adrian Palacios <73246657+adpaco-aws@users.noreply.github.com>
Co-authored-by: Adrian Palacios <73246657+adpaco-aws@users.noreply.github.com>
Co-authored-by: Adrian Palacios <73246657+adpaco-aws@users.noreply.github.com>
celinval
left a comment
There was a problem hiding this comment.
It looks good. Just please remove the remaining line numbers from the expected tests.
| |\ | ||
| 5 | #[kani::solver(123)]\ | ||
| | ^^^^^^^^^^^^^^^^^^^^ |
There was a problem hiding this comment.
Removing the line number:
| |\ | |
| 5 | #[kani::solver(123)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^ | |
| |\ | |
| | #[kani::solver(123)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^ |
| |\ | ||
| 5 | #[kani::solver(kissat, minisat)]\ | ||
| | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
There was a problem hiding this comment.
| |\ | |
| 5 | #[kani::solver(kissat, minisat)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| |\ | |
| | #[kani::solver(kissat, minisat)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| |\ | ||
| 5 | #[kani::solver(kissat)]\ | ||
| | ^^^^^^^^^^^^^^^^^^^^^^^ |
There was a problem hiding this comment.
| |\ | |
| 5 | #[kani::solver(kissat)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^^^^ | |
| |\ | |
| | #[kani::solver(kissat)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^^^^ |
| |\ | ||
| 5 | #[kani::solver]\ | ||
| | ^^^^^^^^^^^^^^^ |
There was a problem hiding this comment.
| |\ | |
| 5 | #[kani::solver]\ | |
| | ^^^^^^^^^^^^^^^ | |
| |\ | |
| | #[kani::solver]\ | |
| | ^^^^^^^^^^^^^^^ |
| |\ | ||
| 5 | #[kani::solver(foo)]\ | ||
| | ^^^^^^^^^^^^^^^^^^^^ |
There was a problem hiding this comment.
| |\ | |
| 5 | #[kani::solver(foo)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^ | |
| |\ | |
| | #[kani::solver(foo)]\ | |
| | ^^^^^^^^^^^^^^^^^^^^ |
Description of changes:
Adds two methods that allow the user to select the solver to use with CBMC:
--solveroptionkani::solverattributeThe
kani::solverattribute allows selecting a different solver for each harness, e.g.If the command-line
--solveroption is specified, it overrides the solver specified in the attribute (if any).Valid values are
kissatandminisat. Passing a custom SAT solver binary can be done with:This case gets passed to CBMC's
--external-sat-solveroption directly.Resolved issues:
Resolves #1656
Related RFC:
Optional #ISSUE-NUMBER.
Call-outs:
Testing:
How is this change tested? Added several tests
Is this a refactor change? No
Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.