Allow overriding compile-time test environment paths for relocated test execution

[elyouG]

Problem

Cargo synthesizes several compile-time environment variables for test and bench targets. including:

• OUT_DIR
• CARGO_MANIFEST_DIR
• CARGO_BIN_EXE_

These values work well when tests are compiled and executed within Cargo’s normal build tree.
The problem I ran into is with Rust tests executed through Yocto ptest.
In that setup, test binaries are typically built in Cargo’s build tree, then installed later into a packaged test directory and executed from there. In that situation, the original compile-time paths embedded by Cargo may no longer match the runtime layout:

• OUT_DIR may point to a build directory that no longer exists on the target
• CARGO_MANIFEST_DIR may no longer be the right base path for fixtures or installed test resources
• CARGO_BIN_EXE_ may need to point to an installed binary location instead of a build-tree-derived path

As a result, some tests fail even though the tested code itself is correct.
I also do not appear to be the first one to run into this class of issue. For example, a PR was proposed in rpm-sequoia to work around the same kind of problem by introducing project-specific environment variables for tests in a Yocto / cross-compilation context:

• see Use optional env vars to force runtime paths in tests rpm-software-management/rpm-sequoia#86

What led me here is that this is not isolated to a single package. I am seeing multiple Rust packages that behave similarly, and I would like to avoid solving the same problem crate-by-crate when the affected values are already synthesized by Cargo itself.

Proposed Solution

Add opt-in overrides for these test/bench compile-time values in artifact.rs::get_env():

• __CARGO_TEST_OUT_DIR_OVERRIDE
• __CARGO_TEST_MANIFEST_DIR_OVERRIDE
• __CARGO_TEST_BIN_EXE_DIR_OVERRIDE

Behavior would be:

• only apply to test and bench units
• only apply when the corresponding __CARGO_TEST_* variable is explicitly set
• preserve current Cargo behavior otherwise
• preserve the existing cargo check fallback behavior for CARGO_BIN_EXE_*

The idea is to let environments such as Yocto ptest adapt Cargo’s standard compile-time test paths directly, instead of requiring each affected crate to implement its own workaround.
For example, in the kind of Yocto setup that motivated this issue, this would make it possible to adapt the test environment directly in the recipe, for example:

do_compile_ptest_cargo:prepend() {
    os.environ["__CARGO_TEST_OUT_DIR_OVERRIDE"] = d.getVar("libdir")
    os.environ["__CARGO_TEST_MANIFEST_DIR_OVERRIDE"] = d.getVar("PTEST_PATH")
}

do_install_ptest:append () {
    install -d ${D}${PTEST_PATH}/src
    install -m 644 ${S}/src/symbols.txt ${D}${PTEST_PATH}/src/symbols.txt
}

Notes

I have already opened a PR for this change, but I moved it back to draft while waiting for feedback and agreement from reviewers on this issue and on the proposed direction.
Thanks in advance to the maintainers and reviewers for taking the time to look at this. Please feel free to share any feedback, questions, or suggestions.

Cc: @ycongal-smile

[weihanglo]

The problem I ran into is with Rust tests executed through Yocto ptest.

#3670 is the parent issue for all the use cases that want to run test executables standalone.

[elyouG]

The problem I ran into is with Rust tests executed through Yocto ptest.

#3670 is the parent issue for all the use cases that want to run test executables standalone.

Thanks for the pointer.
Should I add a short explanation of my use case directly on #3670, or is having my issue linked from there enough?

[epage]

@weihanglo while the title for #3670 is general, at least from the issue text is focused purely on accessing the test binaries themselves. I would keep these as distinct since almost all of the discussion would be independent.

[epage]

__CARGO_* environment variables are internal-only variables, so if we did this, it wouldn't be with those names.

OUT_DIR is more specifically for accessing the results of build scripts. I would worry about accessing this at runtime within a non-local test and would want to know more about the use cases.

MANIFEST_DIR is for one specific package while cargo test can run for many packages.

Ditto for CARGO_BIN_EXE_*. These also have an alternative solution: we now expose these variables at runtime to the tests. If you contribute upstream patches to switch to the runtime versions, then you can pass it in yourself and it will work.

I wonder if we should evaluate passing in more variables at runtime. The weird thing is when should a transitive dependency be able to access a package-specific or binary-specific variable, like MANIFEST_DIR?

[weihanglo]

@weihanglo while the title for #3670 is general, at least from the issue text is focused purely on accessing the test binaries themselves. I would keep these as distinct since almost all of the discussion would be independent.

Yeah that is why I called it parent issue not duplicate. That one also talks about library search path which I assume it is one of the reason people wanting to access OUT_DIR.

I wonder where these compile time overrides can be done in the test executable itself, and falls back to Cargo's envs when missing.

Edit: The linked PR seems done it in test executable already

[weihanglo]

I wonder if we should evaluate passing in more variables at runtime. The weird thing is when should a transitive dependency be able to access a package-specific or binary-specific variable, like MANIFEST_DIR?

From the context of this issue, the motivation of it is to run test executable standalone I guess? I think before adding more runtime envs, we'll need to consider at what level Cargo supports the use case. We called out some drawbacks in the PR of CARGO_BIN_EXE_*: #16421.

[elyouG]

Thanks, this is very useful feedback.

I also have doubts about the __CARGO_TEST_* names. I used them only as draft names for an opt-in mechanism on the Cargo side, not because I consider them the right public interface. If we converge on a solution here, I am absolutely open to other naming proposals.

To give more context, the use case that motivated this work is Yocto ptest, together with ptest-cargo, in a cross-compilation setup.

In that environment, tests are not compiled and executed in the same place:

• they are compiled on the build host, in Cargo's build tree
• then installed into a target image
• and later executed from an installed ptest directory on the target

So the runtime environment is not the same as the compilation environment, and paths captured at compile time may no longer be valid when the test executable is actually run.
For OUT_DIR, the concrete example that led me here is rpm-sequoia
In tests/symbols.rs, it does:
let out_dir = PathBuf::from(env!("OUT_DIR"));

and then walks upwards from that path until it finds librpm_sequoia.so.

In a normal local Cargo build, that works because OUT_DIR points into Cargo's build tree and the built library is still reachable from there.

In the Yocto ptest case, however, env!("OUT_DIR") gets compiled as a path inside Yocto's build TMPDIR, i.e. a host-side build path. Once the test is installed and run on the target, that path no longer refers to a valid runtime location. In that specific case, the test is looking for a library, so the downstream workaround is to override OUT_DIR to Yocto's installed libdir, which is where the library is actually available at runtime.

So the issue for me is not general runtime use of OUT_DIR in ordinary local tests, but specifically that some real tests embed it at compile time and use it later in an installed / relocated execution environment.

For CARGO_MANIFEST_DIR, I understand the concern that cargo test can operate on many packages. In my motivating use case, though, tests are handled package-by-package, not as a multi-package workspace test invocation.

With Yocto ptest, each crate has its own recipe, its own installed test payload, and its own override values if needed. So in practice, the environment is already scoped per package. For example, the same rpm-sequoia test uses:

let mut expected_symbols_txt_fn = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
expected_symbols_txt_fn.push("src/symbols.txt");

and in a packaged ptest setup the original manifest directory from the build host no longer exists on the target. The downstream solution there is to install the needed resource into the ptest area and redirect CARGO_MANIFEST_DIR to that installed location.

I have seen the same kind of usage in deqp-runner, where integration tests also use env!("CARGO_MANIFEST_DIR") to find packaged resources.

So from the perspective of my actual use case, CARGO_MANIFEST_DIR is not acting as a global "this whole cargo test invocation" variable, but as a package-scoped path for the test unit currently being compiled and later installed.

For CARGO_BIN_EXE_*, I did notice the recent runtime work when I started investigating this topic. My initial goal was to see whether Cargo could provide a more general solution on its side for packaged / relocated test execution.

That said, I think your suggestion makes sense, and I would like to evaluate it more concretely on a downstream package. deqp-runner is probably a good candidate here, because it already makes use of env!("CARGO_BIN_EXE_*").

So for this part, I think the next useful step for me is to try the runtime approach on a real downstream case and see whether it works well in practice, including from the Yocto reproducibility point of view. If it does, then I agree that this may be the better direction for binaries, and it could also help encourage downstream crates to standardize on the Cargo-provided runtime mechanism instead of custom path reconstruction.

My original intuition for the override approach was that it felt safer for OUT_DIR / CARGO_MANIFEST_DIR: it extends Cargo with an opt-in way to adapt these values only when an environment really needs it, without changing the default behavior for existing users. In contrast, moving things to runtime means asking downstream projects to change how they consume those values, and I am not yet sure whether that could affect other use cases I have not encountered.

[epage]

So the runtime environment is not the same as the compilation environment, and paths captured at compile time may no longer be valid when the test executable is actually run.
For OUT_DIR, the concrete example that led me here is rpm-sequoia
In tests/symbols.rs, it does:
let out_dir = PathBuf::from(env!("OUT_DIR"));

and then walks upwards from that path until it finds librpm_sequoia.so.

That is relying on internal details of Cargo and us explicitly adding features to support it is close to blessing it. This would likely be better handled by #13663.

Note: these implementation details will be changing in #15010 and may change again in the future (we have concrete ideas for further improvements).

For CARGO_MANIFEST_DIR, I understand the concern that cargo test can operate on many packages. In my motivating use case, though, tests are handled package-by-package, not as a multi-package workspace test invocation.

With Yocto ptest, each crate has its own recipe, its own installed test payload, and its own override values if needed. So in practice, the environment is already scoped per package. For example, the same rpm-sequoia test uses:

That works for yocto but we need to consider the needs of Cargo as a whole and how different features interact with each other.

[elyouG]

So the runtime environment is not the same as the compilation environment, and paths captured at compile time may no longer be valid when the test executable is actually run.
For OUT_DIR, the concrete example that led me here is rpm-sequoia
In tests/symbols.rs, it does:
let out_dir = PathBuf::from(env!("OUT_DIR"));
and then walks upwards from that path until it finds librpm_sequoia.so.

That is relying on internal details of Cargo and us explicitly adding features to support it is close to blessing it. This would likely be better handled by #13663.

Note: these implementation details will be changing in #15010 and may change again in the future (we have concrete ideas for further improvements).

I will take a closer look at #13663 and #15010 to better understand the direction there and how it relates to my case and I will report back once I have done that.

For CARGO_MANIFEST_DIR, I understand the concern that cargo test can operate on many packages. In my motivating use case, though, tests are handled package-by-package, not as a multi-package workspace test invocation.
With Yocto ptest, each crate has its own recipe, its own installed test payload, and its own override values if needed. So in practice, the environment is already scoped per package. For example, the same rpm-sequoia test uses:

That works for yocto but we need to consider the needs of Cargo as a whole and how different features interact with each other.

Would you have suggestions for other directions I could investigate for this specific problem,providing a package-scoped resource path to tests once they are executed outside Cargo's normal environment?

[elyouG]

So the runtime environment is not the same as the compilation environment, and paths captured at compile time may no longer be valid when the test executable is actually run.
For OUT_DIR, the concrete example that led me here is rpm-sequoia
In tests/symbols.rs, it does:
let out_dir = PathBuf::from(env!("OUT_DIR"));
and then walks upwards from that path until it finds librpm_sequoia.so.

That is relying on internal details of Cargo and us explicitly adding features to support it is close to blessing it. This would likely be better handled by #13663.

Note: these implementation details will be changing in #15010 and may change again in the future (we have concrete ideas for further improvements).

For CARGO_MANIFEST_DIR, I understand the concern that cargo test can operate on many packages. In my motivating use case, though, tests are handled package-by-package, not as a multi-package workspace test invocation.
With Yocto ptest, each crate has its own recipe, its own installed test payload, and its own override values if needed. So in practice, the environment is already scoped per package. For example, the same rpm-sequoia test uses:

That works for yocto but we need to consider the needs of Cargo as a whole and how different features interact with each other.

Resolving CARGO_BIN_EXE_ at runtime works for our Yocto use cases. Thanks!

This leaves the second issue: how should integration tests
reliably locate their external resources (test vectors, libs, etc.) ?
Currently, projects use env!("OUT_DIR") or env!("CARGO_MANIFEST_DIR")
as base paths, using .join() to construct absolute paths to their test resources.

What is the official Cargo recommendation for this? Having clear guidance
would really help us convince upstream maintainers to drop these
workarounds and adopt a standard, Cargo-approved approach instead.