Add support for generating perf maps for simple perf profiling

[bnjbvr]

This adds support for a simpler way to profile with perf, using perf maps files. While jitdump support is pretty effective and complete when you have access to DWARF metadata, the perf-inject step may take lots of time, and it is not optimized at all when modules contain lots of functions. (In our embedding, I've stopped perf-inject after it ran for 5 hours and didn't complete, while creating 175K .so objects.) Simplistic support for perf maps enable basic profiling: caller/callee trees, top/bottom time analysis, flamegraphs etc., and works very nicely when the wasm modules have many functions.

I've also tweaked the documentation and tried to make the feature available in the C API too.

I haven't put this behind a feature toggle because this profiling agent does very little, so it's always compiled when running Linux. For consistency I could add one, if you prefer.

[github-actions]

Subscribe to Label Action

cc @peterhuene

Details

This issue or pull request has been labeled: "wasmtime:api", "wasmtime:c-api", "wasmtime:config", "wasmtime:docs"

Thus the following users have been cc'd because of the following labels:

• peterhuene: wasmtime:api, wasmtime:c-api

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

[github-actions]

Label Messager: wasmtime:config

It looks like you are changing Wasmtime's configuration options. Make sure to
complete this check list:

• If you added a new Config method, you wrote extensive documentation for
  it.

  Details

  Our documentation should be of the following form:

  Short, simple summary sentence.
  
  More details. These details can be multiple paragraphs. There should be
  information about not just the method, but its parameters and results as
  well.
  
  Is this method fallible? If so, when can it return an error?
  
  Can this method panic? If so, when does it panic?
  
  # Example
  
  Optional example here.
  

• If you added a new Config method, or modified an existing one, you
  ensured that this configuration is exercised by the fuzz targets.

  Details

  For example, if you expose a new strategy for allocating the next instance
  slot inside the pooling allocator, you should ensure that at least one of our
  fuzz targets exercises that new strategy.

  Often, all that is required of you is to ensure that there is a knob for this
  configuration option in wasmtime_fuzzing::Config (or one
  of its nested structs).

  Rarely, this may require authoring a new fuzz target to specifically test this
  configuration. See our docs on fuzzing for more details.

• If you are enabling a configuration option by default, make sure that it
  has been fuzzed for at least two weeks before turning it on by default.

---

Details

To modify this label's message, edit the .github/label-messager/wasmtime-config.md file.

To add new label messages or remove existing label messages, edit the
.github/label-messager.json configuration file.

Learn more.

[alexcrichton]

Whoa I had no idea this was a supported format by perf! Is there some online documentation for this? Or is there a reason to use jitdump over this?

I've got some minor comments but otherwise this seems reasonable to me to add.

[bjorn3]

Or is there a reason to use jitdump over this?

Jitdump can handle code memory being reused by multiple functions at different times. In addition it allows perf to show the actual instructions that executed rather than just the function names. It also supports recording unwind tables as opposed to requiring frame pointers for unwinding. And finally it allows recording line tables.

Is there some online documentation for this?

https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/jit-interface.txt

[bjorn3]

I think the performance issues of the jitdump support are because we emit a record for each individual function at

wasmtime/crates/jit/src/profiling/jitdump_linux.rs

Line 67 in 76116fc

for (idx, func) in module.finished_functions() {

instead of emitting a single record covering the entire text section of the module as intended.

[bnjbvr]

I think the performance issues of the jitdump support are because we emit a record for each individual function at

Nice find @bjorn3! Although I'm not deep enough into the jitdump support to know how to fix this properly and quickly, and perfmap support is a great start for our use case.

[alexcrichton]

Thanks for the info and the link @bjorn3! I'm not sure what I expected in the way of documentation but... that I suppose at least does what it says on the tin! Makes sense we'll definitely want to keep the jitdump interface for "more robust" usage.

instead of emitting a single record covering the entire text section of the module as intended.

Oh I didn't realize that it was possible to do this, but reading over the documentation it looks like JIT_CODE_LOAD only supports one symbol? Do you know off the top of your head how the whole text section would be loaded? I'd love to fix this myself to avoid generating thousands of tiny *.so files on all perf inject calls.

[alexcrichton]

Ah I missed this earlier, but I think we'll want ot do something other than ignoring errors here ideally. If an error happens it should probably "close" the file and terminate all future writing to it I suspect? (along with perhaps a warning message printed?)

[bnjbvr]

Good point! I've added log messages and early-returns. Looks like a BufWriter will automatically flush in its Drop impl, so we can avoid doing it explicitly in most early-returns.

[alexcrichton]

I think this may want a flush at the end too?

Also with a BufWriter I think you could pass that into make_line to avoid the intermediate string allocation (e.g. write directly to the buffer). Although not required of course, that's ok to defer to if it's ever actually an issue.

[bnjbvr]

Turns out File is also a Write implementator, so could have make_line take &mut dyn Write and use it here without having a BufWriter (since there's only one write in this function, it didn't seem worth having an extra buffer).

[bjorn3]

Oh I didn't realize that it was possible to do this, but reading over the documentation it looks like JIT_CODE_LOAD only supports one symbol? Do you know off the top of your head how the whole text section would be loaded? I'd love to fix this myself to avoid generating thousands of tiny *.so files on all perf inject calls.

Right, I should have read the docs more carefully. My bad. All we can do I guess is to make perf inject not produce a .so for each individual function or write some rust code that has the same functionality as perf inject but is faster.

[bjorn3]

Wow, perf inject has a pretty naive implementation. It is not even that we do something wrong causing each function to be a separate elf file. Instead it really produces a elf file for each function unconditionally: https://github.com/torvalds/linux/blob/9c1bec9c0b08abeac72ed6214b723adc224013bf/tools/perf/util/jitdump.c#L466

[bnjbvr]

or write some rust code that has the same functionality as perf inject but is faster.

👀 mstange/samply#33

[bnjbvr]

I should have addressed all the feedback, thanks for the review!