tests.mk/runtest.rs splits RUSTFLAGS/host_rustcflags,target_rustcflags inappropriately

[infinity0]

edit: TL;DR: tests.mk and runtest.rs respectively split RUSTFLAGS and {host,target}_rustcflags inappropriately; reproduce this in rust upstream (1.8 and earlier) by:

• test.mk bug: set RUSTFLAGS="-C link-args=\"-Wl,-Bsymbolic-functions -Wl,-z,relro\"" when building tests
• runtest.rs bug: set RUSTFLAGS="-C link-args='-Wl,-Bsymbolic-functions -Wl,-z,relro'" when running tests.

These are two separate and independent bugs.

---

Hi, I'm having a hard time debugging a build failure and was wondering if anyone could help. We have multiple identical build failures for Ubuntu across different versions of rustc e.g. see 1.5 on amd64-xenial, 1.8-nightly-20160209 on i386-xenial, 1.8-nightly-20160209 on amd64-xenial. The packages are based on the working Debian packages.

The failure looks related to the command line being run; however they are very similar on Debian and Ubuntu (newlines manually inserted for clarity):

--- debian (from my local testing)
+++ ubuntu (from the 1.8 amd64-xenial link above)
@@ -1,41 +1,41 @@
 LD_LIBRARY_PATH=
  /«BUILDDIR»/rustc-1.8.0~~nightly.20160209+dfsg1/x86_64-unknown-linux-gnu/stage2/lib:
  /usr/lib/llvm-3.7/lib:
  $LD_LIBRARY_PATH
 x86_64-unknown-linux-gnu/stage2/bin/compiletest
  --compile-lib-path x86_64-unknown-linux-gnu/stage2/lib
  --run-lib-path x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib
  --rustc-path x86_64-unknown-linux-gnu/stage2/bin/rustc
  --rustdoc-path x86_64-unknown-linux-gnu/stage2/bin/rustdoc
  --llvm-bin-path /usr/lib/llvm-3.7/bin
  --aux-base /«BUILDDIR»/rustc-1.8.0~~nightly.20160209+dfsg1/src/test/auxiliary/
  --stage-id stage2-x86_64-unknown-linux-gnu
  --target x86_64-unknown-linux-gnu
  --host x86_64-unknown-linux-gnu
  --python "/usr/bin/python2.7"
- --gdb-version="GNU gdb (Debian 7.10-1+b1) 7.10"
+ --gdb-version="GNU gdb (Ubuntu 7.10.1-0ubuntu1) 7.10.1"
  --lldb-version=""
  --android-cross-path=/opt/ndk_standalone
  --adb-path=
  --adb-test-dir=
- --host-rustcflags " -C link-args="-Wl,-z,relro" --cfg rtopt -C rpath -O -L x86_64-unknown-linux-gnu/rt"
+ --host-rustcflags " -C link-args="-Wl,-Bsymbolic-functions -Wl,-z,relro" --cfg rtopt -C rpath -O -L x86_64-unknown-linux-gnu/rt"
  --lldb-python-dir=
- --target-rustcflags " -C link-args="-Wl,-z,relro" --cfg rtopt -C rpath -O -L x86_64-unknown-linux-gnu/rt"
+ --target-rustcflags " -C link-args="-Wl,-Bsymbolic-functions -Wl,-z,relro" --cfg rtopt -C rpath -O -L x86_64-unknown-linux-gnu/rt"
  --verbose
  --valgrind-path ""/usr/bin/valgrind"
   --error-exitcode=100
   --fair-sched=try
   --quiet
   --soname-synonyms=somalloc=NONE
   --suppressions=/«BUILDDIR»/rustc-1.8.0~~nightly.20160209+dfsg1/src/etc/x86.supp
   --tool=memcheck
   --leak-check=full"
  --force-valgrind
  --src-base /«BUILDDIR»/rustc-1.8.0~~nightly.20160209+dfsg1/src/test/rustdoc/
  --build-base x86_64-unknown-linux-gnu/test/rustdoc/
  --mode rustdoc
  --logfile tmp/check-stage2-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-rustdocck.log &&
 touch -r
  tmp/check-stage2-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-rustdocck.ok.start_time
  tmp/check-stage2-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-rustdocck.ok &&
 rm tmp/check-stage2-T-x86_64-unknown-linux-gnu-H-x86_64-unknown-linux-gnu-rustdocck.ok.start_time

The failure on Ubuntu looks like this:

run rustdocck [i686-unknown-linux-gnu]: i686-unknown-linux-gnu/stage2/bin/compiletest
thread '<main>' panicked at 'UnrecognizedOption("W")', /«BUILDDIR»/rustc-1.8.0~~nightly.20160209+dfsg1/src/compiletest/compiletest.rs:101
stack backtrace:
   1: 0xf737b154 - sys::backtrace::tracing::imp::write::h59a57150de078a41Btu
   2: 0xf7383891 - panicking::default_handler::_$u7b$$u7b$closure$u7d$$u7d$::closure.43171
   3: 0xf7383412 - panicking::default_handler::he7b491197c6c99803Wy
   4: 0xf7347e85 - sys_common::unwind::begin_unwind_inner::h0ac7a886eb1f5e41qit
   5: 0xf7348937 - sys_common::unwind::begin_unwind_fmt::h5539aae7d4657c99wht
   6: 0xf76e23cd - parse_config::hd63e903e447ce232uCd
   7: 0xf76da65c - main::hcc56979a8470e836GBd
   8: 0xf7382e8a - sys_common::unwind::try::try_fn::h1480467219455099781
   9: 0xf7378a57 - __rust_try
  10: 0xf7382b5c - rt::lang_start::hb58bd81a27354c059Oy
  11: 0xf76e635d - main
  12: 0xf710271d - __libc_start_main
  13: 0xf76d9d10 - <unknown>
/«BUILDDIR»/rustc-1.8.0~~nightly.20160209+dfsg1/mk/tests.mk:743: recipe for target 'tmp/check-stage2-T-i686-unknown-linux-gnu-H-i686-unknown-linux-gnu-rustdocck.ok' failed
make[2]: *** [tmp/check-stage2-T-i686-unknown-linux-gnu-H-i686-unknown-linux-gnu-rustdocck.ok] Error 101

[infinity0]

Hm, it could be due to the double-quoting of {host,target}-rustcflags. On Ubuntu, that would get broken by the shell into two separate options to compiletest:

--target-rustcflags " -C link-args="-Wl,-Bsymbolic-functions
-Wl,-z,relro" --cfg rtopt -C rpath -O -L x86_64-unknown-linux-gnu/rt"

[infinity0]

Oh, looks like this might be the fault of the Debian packaging - we have RUSTFLAGS = -C link-args="$(LDFLAGS)" somewhere near the top. I'll change those to single quotes and try again. Sorry for the noise.

[infinity0]

OK, that temporarily fixed the compile but then there are other issues. Actually there is a bug in rust, in runtest.rs.

The build failures are now like this:

---- [rustdoc] rustdoc/default-impl.rs stdout ----


executing x86_64-unknown-linux-gnu/stage2/bin/rustc /«BUILDDIR»/rustc-1.8.0~~nightly.20160209+dfsg1/src/test/auxiliary/rustdoc-default-impl.rs -L x86_64-unknown-linux-gnu/test/rustdoc/ --target=x86_64-unknown-linux-gnu --crate-type=dylib -L x86_64-unknown-linux-gnu/test/rustdoc/default-impl.stage2-x86_64-unknown-linux-gnu.rustdoc.libaux -C prefer-dynamic --out-dir x86_64-unknown-linux-gnu/test/rustdoc/default-impl.stage2-x86_64-unknown-linux-gnu.rustdoc.libaux -C link-args='-Wl,-Bsymbolic-functions -Wl,-z,relro' --cfg rtopt -C rpath -O -L x86_64-unknown-linux-gnu/rt
------stdout------------------------------

------stderr------------------------------
error: unknown lint: `l,_z,relro'`
note: requested on the command line with `-W l,_z,relro'`
error: aborting due to previous error

------------------------------------------

This suggests that rustc is being passed the wrong flags. And indeed it is:

# src/compiletest/runtest.rs
fn split_maybe_args(argstr: &Option<String>) -> Vec<String> {
    match *argstr {
        Some(ref s) => {
            s
             .split(' ')

This is called several times (in the same file) as split_maybe_args(&config.{host,target}_rustcflags) and these invocations are not appropriate (other invocations are fine). To fix, you will need to change this to split_shell_args(&config.{host,target}_rustcflags), and implement that function of course.

[infinity0]

Ideally, one should also be able to use double quotes in RUSTFLAGS (like matching behaviour for CXXFLAGS, etc), and to fix this you would fix these lines in mk/tests.mk:

        --host-rustcflags "[..]" \
        --lldb-python-dir=$(CFG_LLDB_PYTHON_DIR) \
        --target-rustcflags "[..]" \

One would shell-quote the args in "[..]", e.g. by using the quote bash utility.

[alexcrichton]

Yeah right now most of our tools don't deal well with quoted or sub-quoted arguments. I'm sure the makefiles are a mess in terms of how they pass variables to each tool, and then all our tools currently just split on spaces rather than attempting to interpret quotes, quote escapes, etc.

Seem like a legit bug!

[infinity0]

(edit: ignore my previous comment, it was because the mips parts of the Makefiles were missing, and not because of the issue described here.)

[alexcrichton]

Make's gone now, so closing.