Reuse cargo's binary selection in `cargo miri`

[RalfJung]

Currently, cargo miri run will run all the binaries in the project (ouch) and cargo miri test will test all binaries and libraries. This leads to issues like #700 (we run tests even when they are disabled), and it means one cannot run examples with Miri.

We should find a way to re-use cargo's selection of binaries, tests and so on.

[RalfJung]

Related: #1001

[jonhoo]

To add to this, I think it'd be helpful to support all the various target specifiers that cargo build (and cargo test) supports. A lightly edited list:

$ cargo help build
...
OPTIONS:
    -p, --package <SPEC>...          Package to build (see `cargo help pkgid`)
        --workspace                  Build all packages in the workspace
        --exclude <SPEC>...          Exclude packages from the build
        --lib                        Build only this package's library
        --bin <NAME>...              Build only the specified binary
        --bins                       Build all binaries
        --example <NAME>...          Build only the specified example
        --examples                   Build all examples
        --test <NAME>...             Build only the specified test target
        --tests                      Build all tests
        --bench <NAME>...            Build only the specified bench target
        --benches                    Build all benches
        --all-targets                Build all targets
        --release                    Build artifacts in release mode, with optimizations
        --features <FEATURES>...     Space-separated list of features to activate
        --all-features               Activate all available features
        --no-default-features        Do not activate the `default` feature

All packages in the workspace are built if the `--workspace` flag is supplied. The
`--workspace` flag is automatically assumed for a virtual manifest.
Note that `--exclude` has to be specified in conjunction with the `--workspace` flag.

This may just be a matter of being able to pass custom flags to cargo build, which I don't think is currently possible, no matter how many -- you pass :p

[RalfJung]

This may just be a matter of being able to pass custom flags to cargo build, which I don't think is currently possible, no matter how many -- you pass :p

Maybe... historically we used cargo rustc but that changed (fairly) recently, so maybe resolving this issue here is now easier than before. But currently we still use cargo_metadata to parse the Cargo.toml and get a list of all targets ourselves and then iterate over them.

Basically, all flags before the first -- are meant to go to cargo. But how weird would it be for cargo miri run/cargo miri test to support the flags of cargo build (and not cargo run/cargo test)?

[RalfJung]

Hm, but relying on cargo build to do the enumeration could (a) end up running multiple Miri in parallel (I think you saw that with some of your flag combinations), and (b) make #584 much harder.

[CAD97]

one cannot run examples with Miri

I'm.. having the opposite problem right now. cargo +nightly miri test --all-features --all-targets is running both the tests in my examples (intended) and the examples' main (unintended). The invocation without miri, cargo +nightly test --all-features --all-targets does not run the example main. (This is important for this example, because the main is a REPL, which obviously isn't good for miri on CI.)

I'm currently using a cfg to just replace the example main under miri, but it'd be nice if I didn't have to (and still run the examples' tests under miri).

Here's an example run.

[RalfJung]

That's odd, the match here should only cover tests. My best guess is the extra flags are confusing the cargo check call... I don't know what cargo check --bin <name> --profile test --all-features --all-targets does but it is probably weird.^^

[CAD97]

IIUC, --all-targets overrides any narrowing options. IOW, in the presence of none of --lib, --bin <NAME>..., --bins, example <NAME>..., --examples, --test <NAME>..., --tests, --bench <NAME>..., --benches, or --all-targets, the default targets are built (IIUC, --lib if no bins are present, --bin <name> if a single bin is (or default bin is configured), and error if multiple bins are present (and no default bin is configured).) But if any of the above flags are present, the default targets are not included, and each listed target is added to the list of targets to [check|build|test|run].

So cargo check --lib --profile test --all-targets will add the lib target to the check list and then add all of the targets to the check list. That would also explain why I thought I saw tests multiple times -- because the library tests are actually getting added to the list of targets to compile twice.

[CAD97]

OK, this gets weirder 😅

I set up a dead simple playground with

// src/lib.rs
#[test]
fn test_in_lib() {
    println!("====== test_in_lib =====");
}
// src/main.rs
#[test]
fn test_in_bin() {
    println!("====== test_in_bin =====");
}

fn main() {
    println!("====== main_in_bin =====");
}
// examples/example.rs
#[test]
fn test_in_example() {
    println!("====== test_in_example =====");
}

fn main() {
    println!("====== main_in_example =====");
}

Output:

$ cargo miri test --all-targets
# stdout
running 1 test
test test_in_example ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


running 1 test
test test_in_example ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

# stderr
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.37s
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.37s

$ cargo miri run --all-targets
# stdout

[empty]

# stderr
    Finished dev [unoptimized + debuginfo] target(s) in 0.03s

Rename examples/example.rs to examples/example.rs.ignored

$ cargo miri test --all-targets
# stdout
running 1 test

running 1 test
test test_in_lib ... ok

test test_in_bin ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out


running 1 test

running 1 test
test test_in_lib ... ok
test test_in_bin ... ok


test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

# stderr
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.42s
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished test [unoptimized + debuginfo] target(s) in 1.33s

$ cargo miri run --all-targets
# stdout
====== main_in_bin =====

running 1 test

running 1 test
test test_in_lib ... ok
test test_in_bin ... ok


test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

# stderr
    Checking playground v0.0.0 (D:\repos\cad97\playground)
    Finished dev [unoptimized + debuginfo] target(s) in 1.36s

[CAD97]

One last note: here's proof that the example's main really is definitely running: it hits a panic I added to that main. I don't know why the smaller playground example doesn't seem to be running the example main.

[RalfJung]

I'm rather surprised we do anything with examples, but for now I can only say that --all-targets should not be used with cargo-miri... it violates some assumptions that code is otherwise making.^^
I didn't even know cargo test --all-targets was a thing...