Interrupting engine causes illegal hardware instruction

[EliseZeroTwo]

Environment: rustc 1.50.0-nightly (c609b2eaf 2020-12-20), MacOS, AArch64
Wasmtime Version: 0.21.0
Expected Behavior: Interrupt WASM engine, actually causes zsh: illegal hardware instruction cargo run
To Reproduce: Attempt to run the Store/interrupt_handle example

If there is any more info needed/testing I am here

[bjorn3]

Can you get a backtrace using lldb?

[EliseZeroTwo]

Sure, this is what you wanted right?

elise@Elises-MacBook-Air wasmtime-test % lldb target/debug/wasmtime-test
(lldb) target create "target/debug/wasmtime-test"
Current executable set to '/Users/elise/code/wasmtime-test/target/debug/wasmtime-test' (arm64).
(lldb) run
Process 93711 launched: '/Users/elise/code/wasmtime-test/target/debug/wasmtime-test' (arm64)
Process 93711 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=1, subcode=0xd4a00000)
    frame #0: 0x0000000102af001c
->  0x102af001c: .long  0xd4a00000                ; unknown opcode
    0x102af0020: stp    x29, x30, [sp, #-0x10]!
    0x102af0024: mov    x29, sp
    0x102af0028: blr    x2
Target 0: (wasmtime-test) stopped.
(lldb) bt
error: need to add support for DW_TAG_base_type '()' encoded with DW_ATE = 0x7, bit_size = 0
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_INSTRUCTION (code=1, subcode=0xd4a00000)
  * frame #0: 0x0000000102af001c
    frame #1: 0x0000000100052504 wasmtime-test`wasmtime::func::Func::get0::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hdf065a651b23e6eb at func.rs:219:36
    frame #2: 0x000000010007bdc8 wasmtime-test`wasmtime_runtime::traphandlers::catch_traps::call_closure::h34799d017a830c62(payload="\x88��o") at traphandlers.rs:397:18
    frame #3: 0x0000000100662110 wasmtime-test`RegisterSetjmp(buf_storage=0x000000016fdfef68, body=(wasmtime-test`wasmtime_runtime::traphandlers::catch_traps::call_closure::h34799d017a830c62 at traphandlers.rs:393), payload=0x000000016fdff0e0) at helpers.c:12:3
    frame #4: 0x000000010007c148 wasmtime-test`wasmtime_runtime::traphandlers::catch_traps::_$u7b$$u7b$closure$u7d$$u7d$::h4cbc1bf5cfaf39f6(cx=0x000000016fdfef30) at traphandlers.rs:386:9
    frame #5: 0x000000010007dc5c wasmtime-test`wasmtime_runtime::traphandlers::CallThreadState::with::_$u7b$$u7b$closure$u7d$$u7d$::h43344f8625fc9eb5 at traphandlers.rs:442:38
    frame #6: 0x000000010005e42c wasmtime-test`wasmtime_runtime::traphandlers::tls::set::_$u7b$$u7b$closure$u7d$$u7d$::h32dc2e0d54271bc4(p=0x00000001038082a0) at traphandlers.rs:678:13
    frame #7: 0x00000001000b24dc wasmtime-test`std::thread::local::LocalKey$LT$T$GT$::try_with::h22f85c7b627282cd(self=0x00000001010f9110, f=closure-0 @ 0x000000016fdfe620) at local.rs:272:16
    frame #8: 0x00000001000b2318 wasmtime-test`std::thread::local::LocalKey$LT$T$GT$::with::h0fa96015bd54d6c9(self=0x00000001010f9110, f=<unavailable>) at local.rs:248:9
    frame #9: 0x000000010005e2fc wasmtime-test`wasmtime_runtime::traphandlers::tls::set::h4b883c3c99a44464(ptr=0x000000016fdfef30, closure=closure-0 @ 0x000000016fdfe680) at traphandlers.rs:670:9
    frame #10: 0x000000010007c278 wasmtime-test`wasmtime_runtime::traphandlers::CallThreadState::with::h1b3c574b3e474783(self=CallThreadState @ 0x000000016fdfef30, max_wasm_stack=1048576, closure=closure-0 @ 0x000000016fdfed10) at traphandlers.rs:442:19
    frame #11: 0x000000010007c08c wasmtime-test`wasmtime_runtime::traphandlers::catch_traps::hf821c618b37cd358(vmctx=0x0000000102c08d20, max_wasm_stack=1048576, is_wasm_code=closure-0 @ 0x000000016fdfee38, signal_handler=Option<&Fn<(i32, *const libc::unix::bsd::apple::siginfo_t, *const core::ffi::c_void)>> @ 0x000000016fdfefb8, closure=closure-0 @ 0x000000016fdff0e0) at traphandlers.rs:385:12
    frame #12: 0x0000000100051a08 wasmtime-test`wasmtime::func::invoke_wasm_and_catch_traps::h748a5c3d509c585d(vmctx=0x0000000102c08d20, store=0x000000016fdff498, closure=closure-0 @ 0x000000016fdff1b0) at func.rs:829:9
    frame #13: 0x00000001000525c0 wasmtime-test`wasmtime::func::Func::get0::_$u7b$$u7b$closure$u7d$$u7d$::hb5574d4b37bd571e at func.rs:218:21
    frame #14: 0x0000000100003f3c wasmtime-test`wasmtime_test::main::haebedb672b1ad106 at main.rs:23:16
    frame #15: 0x0000000100001bc0 wasmtime-test`core::ops::function::FnOnce::call_once::h2185be2c6c66daf8((null)=(wasmtime-test`wasmtime_test::main::haebedb672b1ad106 at main.rs:5), (null)=<unavailable>) at function.rs:227:5
    frame #16: 0x00000001000024ac wasmtime-test`std::sys_common::backtrace::__rust_begin_short_backtrace::h015d35dd00b7b79a(f=(wasmtime-test`wasmtime_test::main::haebedb672b1ad106 at main.rs:5)) at backtrace.rs:125:18
    frame #17: 0x0000000100003a7c wasmtime-test`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h272b1220ab4da625 at rt.rs:66:18
    frame #18: 0x0000000100d42dd4 wasmtime-test`std::rt::lang_start_internal::h17327e074e29c857 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::hba3336600b4e2d51 at function.rs:259:13 [opt]
    frame #19: 0x0000000100d42dc8 wasmtime-test`std::rt::lang_start_internal::h17327e074e29c857 [inlined] std::panicking::try::do_call::hb525555825cb795d at panicking.rs:379 [opt]
    frame #20: 0x0000000100d42dc8 wasmtime-test`std::rt::lang_start_internal::h17327e074e29c857 [inlined] std::panicking::try::hd444835db4887372 at panicking.rs:343 [opt]
    frame #21: 0x0000000100d42dc8 wasmtime-test`std::rt::lang_start_internal::h17327e074e29c857 [inlined] std::panic::catch_unwind::h73a34a4a8bd42478 at panic.rs:396 [opt]
    frame #22: 0x0000000100d42dc8 wasmtime-test`std::rt::lang_start_internal::h17327e074e29c857 at rt.rs:51 [opt]
    frame #23: 0x0000000100003a54 wasmtime-test`std::rt::lang_start::hfad81a1e4e1ebb69(main=(wasmtime-test`wasmtime_test::main::haebedb672b1ad106 at main.rs:5), argc=1, argv=0x000000016fdff728) at rt.rs:65:5
    frame #24: 0x000000010000415c wasmtime-test`main + 40
    frame #25: 0x00000001a472cf34 libdyld.dylib`start + 4

[bjorn3]

Thanks. While I am not familiar with this part of the code, this backtrace will likely be useful for someone who is familiar with it.

[cfallin]

Thanks for the report @EliseZeroTwo!

0xd4a00000 is our specially-chosen undefined instruction, borrowed from SpiderMonkey (there is some good discussion in that comment on how to reliably get a SIGILL on arm64). It is how we emit traps (Inst::Udf here).

What seems to be happening is that the SIGILL isn't being caught in this case; so there's something wrong with the signal handling in the case that SIGINT (or whatever Ctrl-C becomes in this case) is being processed.

I recall something recently about the interaction between Mach message ports (on macOS) and POSIX signals, and how there might be an issue there... does anyone else know more? Sadly I don't have an arm64 Mac to test with...

[sunfishcode]

The recent discussion of Mach message ports on macOS is in #2456.

[alexcrichton]

AFAIK we haven't done testing yet on arm64 macOS machines, so this is likely a bug with integration with the new system. This looks like the signal either isn't being caught (e.g. mach messages and such), or it's correctly caught but wasmtime is incorrectly concluding that the signal isn't related to wasm code. In the latter case the signal will look like a normal uncaught signal.

This'd probably need someone with arm64 hardware to debug a bit to figure out what's going on here. Unfortunately I don't myself have such hardware :(

[akirilov-arm]

This is a bit tangential to the issue, but unless I am missing some context, we should be able to switch to the UDF "instruction".

[bjorn3]

As far as I know .long 0xd4a00000 is a variant of the UDF instruction with 0xa0d4 as payload. https://reviews.llvm.org/D53319

[akirilov-arm]

No, UDF is encoded as 0xFFFF or less; in particular, 0 corresponds to UDF #0.

[cfallin]

It looks like @bnjbvr originally wrote the comment in SpiderMonkey above that describes why they chose 0xd4a00000. As I recall, we had originally generated proper UDFs but had to switch for some reason while bringing up SpiderMonkey/Cranelift integration. @bnjbvr, do you have any more details? (It's not hugely important but it would be good to document exactly why we can't use UDF, if that's the case.)

[bnjbvr]

Yes, a few reasons come to mind. None of them seem particularly inevitable, and they could all be worked around, if we decided to use a new value, but it was simpler to choose this one for a few reasons:

• this is the same value Spidermonkey uses for triggering SIGILL, with the reasons explained there: https://searchfox.org/mozilla-central/source/js/src/jit/arm64/vixl/Constants-vixl.h#2668-2690 . If we didn't use the same value as Spidermonkey did use, either Cranelift would need a compiler option to allow selecting which undefined instruction is used (or select it among a list of predefined), or Spidermonkey should refactor its code to support different undefined instruction encodings in their aarch64 backend.
• there might be an argument that a native compiler (rustc/llvm) would use the default UDF instruction, and that using a different instruction in cranelift-generated code makes it easier to spot what's the actual issue in a test case. For instance, for the initial issue discussed here in particular, we could directly infer that it was a cranelift-generated undefined encoding not being properly handled by the OS. A sigill with another undefined encoding could have been this problem or another illegal encoding problem (coming from llvm codegen in native code).

[akirilov-arm]

The issue with the current encoding is that it is undefined now, but may no longer be in the future. UDF on the other hand is permanently undefined, i.e. it can be viewed as a permanent encoding space reservation.

[repi]

We've been running into the same issue in some our WASM modules with wasmtime on aarch64-apple-darwin as well as on aarch64-linux-android