Skip to content

[fuzz] Configure the differential target#4773

Merged
abrown merged 5 commits into
bytecodealliance:mainfrom
abrown:fuzz-lists
Aug 24, 2022
Merged

[fuzz] Configure the differential target#4773
abrown merged 5 commits into
bytecodealliance:mainfrom
abrown:fuzz-lists

Conversation

@abrown
Copy link
Copy Markdown
Member

@abrown abrown commented Aug 24, 2022

This change is a follow-on from #4515 to add the ability to configure
the differential fuzz target by limiting which engines and modules are
used for fuzzing. This is incredibly useful when troubleshooting, e.g.,
when an engine is more prone to failure, we can target that engine
exclusively. The effect of this configuration is visible in the
statistics now printed out from #4739.

Engines are configured using the ALLOWED_ENGINES environment variable.
We can either subtract from the set of allowed engines (e.g.,
ALLOWED_ENGINES=-v8) or build up a set of allowed engines (e.g.,
ALLOWED_ENGINES=wasmi,spec), but not both at the same time.
ALLOWED_ENGINES only configures the left-hand side engine; the
right-hand side is always Wasmtime. When omitted, ALLOWED_ENGINES
defaults to [wasmtime, wasmi, spec, v8].

The generated WebAssembly modules are configured using
ALLOWED_MODULES. This environment variables works the same as above
but the available options are: [wasm-smith, single-inst].

This change is a follow-on from bytecodealliance#4515 to add the ability to configure
the `differential` fuzz target by limiting which engines and modules are
used for fuzzing. This is incredibly useful when troubleshooting, e.g.,
when an engine is more prone to failure, we can target that engine
exclusively. The effect of this configuration is visible in the
statistics now printed out from bytecodealliance#4739.

Engines are configured using the `ALLOWED_ENGINES` environment variable.
We can either subtract from the set of allowed engines (e.g.,
`ALLOWED_ENGINES=-v8`) or build up a set of allowed engines (e.g.,
`ALLOWED_ENGINES=wasmi,spec`), but not both at the same time.
`ALLOWED_ENGINES` only configures the left-hand side engine; the
right-hand side is always Wasmtime. When omitted, `ALLOWED_ENGINES`
defaults to [`wasmtime`, `wasmi`, `spec`, `v8`].

The generated WebAssembly modules are configured using
`ALLOWED_MODULES`. This environment variables works the same as above
but the available options are: [`wasm-smith`, `single-inst`].
@abrown abrown requested a review from jameysharp August 24, 2022 17:55
@abrown
Copy link
Copy Markdown
Member Author

abrown commented Aug 24, 2022

Here is an example from running the target locally:

$ ALLOWED_MODULES=-wasm-smith cargo +nightly fuzz run differentia
...
=== Execution rate (384 successes / 2000 attempted modules): 19.20% ===
        wasmi: 0.96%, spec: 8.85%, wasmtime: 83.97%, v8: 6.22%
        wasm-smith: 0.00%, single-inst: 100.00%

@abrown
Copy link
Copy Markdown
Member Author

abrown commented Aug 24, 2022

@jameysharp, you've had some opinions on this in the past so I tagged you for review (?): I really would have preferred to use &[&str] everywhere (in the static variables, function parameters, etc.) but once env gave me back Strings this all started sliding toward Vec<String>. This makes for some awkward match and if statements. Do you see an easy way to convert to slices and &str or even just one of those?

Copy link
Copy Markdown
Contributor

@jameysharp jameysharp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true: I have, at times, had opinions! 😆

I don't feel very strongly about any of these review comments. This PR is fine as-is, so I'm marking this approved. But if you want to take a little more time on it I think these suggestions are improvements.

Comment thread crates/fuzzing/src/oracles/engine.rs
Comment thread crates/fuzzing/src/oracles/engine.rs Outdated
Comment thread crates/fuzzing/src/oracles/engine.rs
Comment thread fuzz/fuzz_targets/differential.rs Outdated
Comment thread fuzz/fuzz_targets/differential.rs Outdated
Comment thread crates/fuzzing/src/oracles/engine.rs
@github-actions github-actions Bot added the fuzzing Issues related to our fuzzing infrastructure label Aug 24, 2022
@github-actions
Copy link
Copy Markdown

Subscribe to Label Action

cc @fitzgen

Details This issue or pull request has been labeled: "fuzzing"

Thus the following users have been cc'd because of the following labels:

  • fitzgen: fuzzing

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

Comment thread crates/fuzzing/src/oracles/engine.rs Outdated
Copy link
Copy Markdown
Contributor

@jameysharp jameysharp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you've addressed all my suggestions. Thanks! I hope you found these suggestions helpful.

@abrown
Copy link
Copy Markdown
Member Author

abrown commented Aug 24, 2022

I think you've addressed all my suggestions. Thanks! I hope you found these suggestions helpful.

Yeah, they were great!

@abrown abrown merged commit d3c463a into bytecodealliance:main Aug 24, 2022
@abrown abrown deleted the fuzz-lists branch August 24, 2022 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fuzzing Issues related to our fuzzing infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants