Support for Wasm Coredump

[xtuc]

Feature

When the Wasm instance traps, it's sometimes difficult to understand what happened. Post-mortem debugging using coredumps (which is extensively used in native environment) would be helpful for investigating and fixing crashes.

Wasm coredump is especially useful for serverless environment where production binaries are stripped and/or have access to limited logging.

Implementation

Implement Wasm coredumps as specified by https://github.com/WebAssembly/tool-conventions/blob/main/Coredump.md.
Note that the spec is early and subject to changes. Feedback very welcome!

cc @fitzgen

[bjorn3]

Reading the linear memory after a crash is already possible. As for getting the locals and stack values, this is much more complicated. Wasmtime uses the Cranelift optimizing compiler, which can eliminate locals and stack values entirely and leaves those that remain at whichever location it likes. It did be necessary to somehow prevent optimizing locals away, at least for points where a trap could happen. There is debugger support for getting the location of locals and stack values which aren't optimized away to generate debuginfo, but I'm not sure if it is 100% accurate. By the way #5537 is somewhat relevant to this.

[xtuc]

I don't think Wasm coredump should prevent optimizations, given that ideally it's enabled by default.

It's not uncommon to see coredump in native environment with missing values because they were optimized away. They are usually not very helpful for debugging.

[bjorn3]

The wasm coredump format doesn't seem to allow omitting values that are optimized away, but if it is allowed, then it should be possible to implement without too much changes to Cranelift. I think it would need some changes to the unwind table generation code to store the location of callee saved registers, but that will need to be done anyway for handling exceptions. After that I guess it would be a matter of telling Cranelift to generate debuginfo and then during a crash unwind the stack and record all preserved locals and stack values for every frame from Wasmtime.

[xtuc]

The wasm coredump format doesn't seem to allow omitting values that are optimized away

Correct, at the moment it doesn't. I'm going to add it, thanks for your input!

[jameysharp]

This is an area I haven't dug into much, but doesn't Cranelift's support for GC already support tracking the information we need for this? I think we would need to mark potentially-trapping instructions as "safe points" and then request stack maps from Cranelift. And my impression was that calls are already considered safe points. But this is all conjecture based on a CVE that I was peripherally paying attention to last year, so I could have it all wrong.

[fitzgen]

This is an area I haven't dug into much, but doesn't Cranelift's support for GC already support tracking the information we need for this? I think we would need to mark potentially-trapping instructions as "safe points" and then request stack maps from Cranelift. And my impression was that calls are already considered safe points. But this is all conjecture based on a CVE that I was peripherally paying attention to last year, so I could have it all wrong.

Stack maps only track reference values (r32/r64), and only say which stack slots have live references in them. They do not supply any kind of info to help tie that back to wasm locals or even clif SSA variables.

I don't think we would want to use stack maps for this stuff.

[cfallin]

On the flip-side, if you're proposing altering the generated code to assist debugging observability @jameysharp, there is a large design space that we haven't really explored. A relatively simple change would be to define a pseudoinstruction that takes all locals as inputs, with "any" constraints to regalloc (stack slot or register), and insert these wherever a crash could happen. This "state snapshot" instruction would then guarantee observability of all values, at the cost of hindering optimization.

This goes somewhat against the "don't alter what you're observing" principle that is common in debug infrastructure but I'll note that we do already have some hacks to keep important values alive (in this case, the vmctx, which makes all other wasm state reachable) for the whole function body.

There's also the "recovery instruction" approach, used in IonMonkey at least: whenever a value is optimized out, generate a side-sequence of instructions that can recompute it. That's a much larger compiler-infrastructure undertaking but in principle we could do it, if perfect debug observability were a goal.

[xtuc]

WebAssembly/tool-conventions#198 has been closed. The coredump format now allows to mark local/stack values as missing.

[xtuc]

I made a change to add initial/basic coredump generation: #5868. Could you please have a look and let me know if this is the right direction?
It uses WasmBacktrace for information about frames.