cdylib / staticlib libraries aren't created in the target directory when running `cargo test`

[jgallagher-cs]

Problem

I'm not 100% sure this is a bug, and as an attempt to avoid an X/Y problem, I'll start by describing what I'm ultimately trying to do (which is related to #7825).

I have Rust crates with crate-type = ["cdylib"] or crate-type = ["staticlib"] (or both), and I'd like to run what are essentially integration tests written in C to ensure that my header files are correct, the libraries are built correctly, etc. And I'd really like for those tests to run when I run cargo test.

What I'm currently doing which almost works is that I have two crates: one that produces the C library and the other that exists solely to run the integration tests. The testing crate uses cc to build C code that calls the library produced by the "real" crate; its build.rs looks roughly like this (this example is assuming a staticlib):

fn main() {
    cc::Build::new()
        .file("src/tests.c")
        .include("../real-crate/include")
        .compile("c_integration_tests");

    let profile_dir = /* ... */; // find "target/release" or "target/debug" based on OUT_DIR
    println!("cargo:rustc-link-search=native={}", profile_dir);
    println!("cargo:rustc-link-lib=static=real_crate");
}

There are (at least) two problems with this:

• Finding target/release or target/debug based on OUT_DIR is pretty hacky.
• target/release/libreal_crate.a isn't actually built when I run cargo test (even if it's a dependency); only the hash-suffixed library name (target/release/deps/libreal_crate-XXXXXXXXXXXXXXX.a) exists.

The first problem is a little gross but hasn't seemed to be a big deal in practice, although I assume it's something that might change over time. The second is more problematic; a workaround is to cargo build before cargo testing, but it's hard to remember to do that every time, and it's not ideal to have to tell everyone working on the project that that's a requirement.

Steps

1. Create a crate with crate-type = ["staticlib"].
2. Create another crate that depends on it.
3. Run cargo test.
4. The static lib from the first crate is not built in the target dir.

Possible Solution(s)

I considered finding target/{release,debug}/deps via OUT_DIR then scanning that directory to find the library-with-suffix and linking that, but that seems even more fragile.

Notes

Output of cargo version:

cargo 1.43.0 (3532cf7 2020-03-17)

[joshtriplett]

This makes sense. In general, there's a use case for depending on a crate that provides a cdylib, a staticlib, or even a bin, and then being able to use the resulting artifact at build time.

We discussed this in the Cargo meeting today.

I'd propose the following (sketch of a) solution:

• Cargo.toml should allow specifying dependencies (or build dependencies, or dev dependencies) on a staticlib or cdylib or bin.
  • This likely will require some additional indication of what type(s) of artifact the dependency wants.
  • We need to carefully think about whether the type of dependency fully specifies whether you want it built for the host or the target, or if we need an explicit way to specify this.
  • We need to consider the possibility of allowing flexible selection between static and dynamic libraries, if a crate can support building either or both.
• Given such a dependency, Cargo will build the artifacts you asked for, in your output directory.
• Cargo will then export environment variables pointing to those artifacts, along the lines of CARGO_DEP_<depname>_BIN, CARGO_DEP_<depname>_STATICLIB, or CARGO_DEP_<depname>_CDYLIB.

[jgallagher-cs]

Cool! A couple thoughts on your solution sketch:

• Does the "caller" need to specify which kind of artifact(s) it wants, or should that be pulled from the dependency itself? At a minimum, the caller can't ask for a cdylib or staticlib if those aren't in the dependency's crate-type, right?
• I hadn't considered depending on bins but what if the dependency has more than one?

I don't have any experience with cross compilation, so can't comment on the host vs target question.

[asomers]

I'm experiencing this exact same problem. I want to write an integration test for a cdylib. I can guess the built file's name easily enough, but I can't figure out how to force cargo to build it before (or while) running the test. Is there any better workaround than "run cargo build before cargo test"?

[dkg]

Any word on this? I'd like to be able to automate these kinds of integration tests in a cdylib crate as well (see integritychain/fips203#13)

[HoneyPony]

Hi, just wanted to mention my use case for this in case it is helpful designwise (and to express my interest in this functionality).

I am currently using Rust to write a little programming language, PonieScript. I have been using Rust to write the compiler, and now I am (trying to) use Rust to implement the language runtime. (The linked branch is the branch with this Rust runtime).

The tests for my programming language are almost all integration tests. They work by invoking the compiler (retrieved through env!("CARGO_BIN_EXE_poniescript")).

However, in order to properly run the integration tests with the new Rust runtime, I need to be able to both 1) build the runtime library, and 2) locate it. Right now doing both does not seem to be possible.

What I have tried so far is adding the runtime library as a dependency (or dev-dependency or build-dependency) of my poniescript crate (the compiler). This does appear to make cargo build the runtime library; if I run a find target -name "*.a" after building the compiler, I do find a relevant static library:

$ find target -name "*.a"
target/debug/build/libmimalloc-sys-3bc419ed779d236d/out/libmimalloc.a
target/debug/deps/libponiescript_rt-68290b9913e87f35.a # <-- this one
target/debug/deps/libponiescript_gc-a19e6764a709ebe4.a

But from what I can tell, there is no e.g. environment variable that tells me where the library is. I could be mistaken, but I think there are occasionally multiple libponiescript_rt-<numbers>.a files created if I build every crate in the workspace, so I don't think it would work to just search the target/ folder for a relevant-looking crate; I might find one out of date.

Instead, what my integration tester currently assumes is that there will be a libponiescript_rt.a crate in the same directory as the file that I get from env!("CARGO_BIN_EXE_poniescript"):

// We need to link against the poniescript-rt library, written in Rust.
// For now, assume it is next to the poniescript executable, but it would
// be nice to make this better.
let bin_dir = PathBuf::from(env!("CARGO_BIN_EXE_poniescript"));
let bin_dir = bin_dir.parent().unwrap();

At the time I wrote this code, I think I briefly looked into a better way to do this, but didn't find anything at the time, as you can see by my comment there.

This does work in the sense that it lets me run my integration tests, if I first manually ensure the poniescript-rt library is up to date by running cargo build -p poniescript-rt.

All of this is to say that either of the two designs would work for my use case:

1. Probably the best: Give some way to locate cargo artifacts like staticlibs through an environment variable, similarly to CARGO_BIN_EXE_<package>. In particular, I think for my use case I will end up wanting to e.g. have certain features in my poniescript-rt library, and I want the integration tests to be compiled against the version built with those features. I assume that different compiles of the same library with different features end up at different locations right now? If so, this sort of environment variable would be the best way to make things fit together smoothly, I think.
2. Probably worse: Simply let me tell Cargo one way or another to put the output libponiescript_rt.a where I expect it, i.e. in the same directory as my existing CARGO_BIN_EXE_poniescript. I think this seems a bit more fragile, however, and I'm not sure how it would really interact with e.g. compiling the same package with multiple sets of features. But, it would solve my use case at least.

[epage]

Looks like the proposal here was "artifact deps" which has been implemented but not stabilized since the last update, see https://doc.rust-lang.org/cargo/reference/unstable.html#artifact-dependencies

Stabilization is being tracked in #9096.