embedded support

[yamt]

my co-worker told me that wasmtime project wanted to hear use-cases/demands/requirements for embedded scenarios.
here is FYI about our requirements.

Feature

support embedded environment

by "embedded", i meant something like

• 32-bit
• no mmu
• no linux. probably nuttx or freertos.
• memory: from a few hundred KB to several MB (for the entire system, not just wasmtime)
• aot is a must as jit is unrealistic
• storage: probably a several MB
• cpu arch: probably xtensa or 32-bit riscv

Benefit

more adaptation?

Implementation

i myself have no idea.
do you think it's realistic to support this sort of small environments?

Alternatives

• stick with "huge" linux environments like raspberry pi.
• use other runtime. eg. wamr.

[alexcrichton]

Thanks for writing this up! Wasmtime has aways to go to support systems like this, but at least in my opinion it's tractable. Some questions for you:

• Do you know if Rust code runs at all on this system? For example what Rust targets should we look at as a compilation target? I notice, for example, that Rust has a riscv32i-unknown-none-elf target but there's no corresponding Linux target, so it may be that Rust doesn't have a target for this yet (that's ok though). I'm not familiar with xtensa, though, is that an ARM platform?
• Do these environments have a kernel? Or without an MMU is the program basically running directly on the hardware?
• In terms of memory allocation, is there something that's mmap-like or is the thinking that it's "just call malloc"? For example how are linear memories allocated today for wasm?

I think the two major lifts for supporting a platform like this will be (a) a riscv32 backend in Cranelift and (b) porting work to get Wasmtime to compile on the target. I know (a) is a big chunk of work but (b) depends on whether there's a Rust target already for this platform. If there's not it'll be a bit more work.

[cfallin]

Xtensa is a separate ISA. It's a little unique (register windows automatically shifted by calls, deja vu from SPARC!) but it's conceivable it could be supported by Cranelift. If we had to pick one "embedded-friendly" ISA to support I think I'd definitely prefer it be RV32I: it's more "standard" w.r.t. our other ISAs, especially as we support RV64GC already.

My main concern with both, in terms of porting effort, would actually be floating-point: from the above it doesn't look like Xtensa has hardware FP, and RV32I doesn't either; so we'd need to do softfloat (an integer-only Wasm subset would probably not be something we'd ship as it wouldn't be a conforming implementation). We could turn every FP op into a libcall and use an existing softfloat library, I guess.

[bjorn3]

I don't think you would want to avoid floats entirely on an embedded system without fpu as softfloat is expensive, especially on microcontrollers that are already slow. A softfloat impl in wasmtime would also take up a lot of precious space. Entirely omitting float support from wasmtime if the cpu doesn't support it makes sense from that perspective even if not conforming with the wasm spec. Something that would be possible though I think is to have a tool rewriting wasm to replace floats with a softfloat impl inside wasm itself. This can then allow wasm-opt to strip out unused softfloat intrinsics as well as inline those that are used whenever it saves code size.

[alexcrichton]

One possible middle ground related to floats, we could expose wasmparser's floats feature which, if disabled in Wasmtime, would statically verify the input module has no floats anywhere in it. We could then relax the base required feature set for RISC-V to not require the d and f extensions. Optionally we could go further and add has_f and has_d to all of the various instructions from those extensions to surface panic if there's bugs anywhere by accident and anything slips through.

[alexcrichton]

Er to elaborate further on that, we would avoid the need for softfloat anywhere because wasm floats would always compile to hardware floats and we would statically know via verification that the input module doesn't have floats so there's no need for the runtime or the compiler to have a soft-float path.

The only trickiness would be what if the input module had floats, and for that I think @bjorn3's suggestion is a good one, a theoretical pass to insert soft-float routines in the wasm module itself.

[ayakoakasaka]

Thank you for considering the embedded scenario.
Is it possible to gradually move the target closer to the Embedded world?

Do you know if Rust code runs at all on this system? For example what Rust targets should we look at as a compilation target? I notice, for example, that Rust has a riscv32i-unknown-none-elf target but there's no corresponding Linux target, so it may be that Rust doesn't have a target for this yet (that's ok though). I'm not familiar with xtensa, though, is that an ARM platform?

-> Q1

Do these environments have a kernel? Or without an MMU is the program basically running directly on the hardware?

-> Q2

In terms of memory allocation, is there something that's mmap-like or is the thinking that it's "just call malloc"? For example how are linear memories allocated today for wasm?

-> Q3

Phase 1:
Use https://wiki.sipeed.com/hardware/en/maix/m1s/m1s_dock.html
Q1: RV64GCV with https://github.com/bouffalolab/bl-pac/tree/main/bl808 might be used
Q2: the main CPU has MMU and has linux
Q3: just call malloc

Phase 2:
Use https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32c6/esp32-c6-devkitc-1/index.html
Q1: 32-bit RISC-V with https://github.com/esp-rs/esp-rust-board might be used
Q2: no MMU/ no linux
Q3: just call malloc(I guess)

[afonso360]

Phase 1

This should work today! It is pretty much the same as a regular environment, just with a slower CPU and less RAM, so as long as you don't run into those issues you should be good.

It looks like the BL808 has a fairly large 64MiB of RAM. I have never measured how much RAM we use in Wasmtime so I don't know if that is enough.

Phase 2

Looking at the datasheet for that SOC it looks like the CPU is RV32IMAC, so we do have the issues above with not supporting floats / doubles. (And not having a cranelift backend for RISC-V 32)

Having malloc support is good, in that it brings us closer to normal non embedded rust, but I think we'd still need some minimal support for whatever OS is running. (i.e. using Espressif's SDK's I think it's a fork of FreeRTOS)

If you are running without any OS or other platform, then I'm not entirely sure how rust works on those platforms W.R.T threading / interrupts / etc..

[alexcrichton]

I'd echo what @afonso360 said already, and I'd additionally add though that I think it makes sense to move Wasmtime in the direction of embedded. This will require a nontrivial amount of work that will take some time but at least in my opinion it's possible so long as there are resources to apply.

The big-ticket items of RV32 and size optimizations are fairly separable and I suspect there's probably other smaller porting work and such to happen as well. No need for that to all happen as one big bang and I think PRs can trickle in over time to improve the situation here.

[ayakoakasaka]

Thank you for considering embedded systems. As @alexcrichton and @bjorn3 mentioned, soft-floats are too expensive for a weak environment. Therefore, we use hardware floats for our case.

However, I'm grateful to know there's a way even without hardware support.

[TheQuantumPhysicist]

The fact that armv7/armhf are not supported is very disappointing. This is a problem for me because many embedded devices are 32-bit, including Raspberry Pi zero.

Raspberry Pi zero 2 isn't properly supported in many systems, like Buildroot, which makes dealing with 32-bit much easier.

Supporting 32-bit just makes everything much simpler for embedded devices.

[cfallin]

@TheQuantumPhysicist we're well aware! No one in the project has really made a decision not to support 32-bit targets. The reality is rather that it takes a good amount of effort to add a new compiler target (i.e., write a backend), on the order of a few months of full-time work, and none of us can afford that time away from other tasks that we also have to do at the moment.

Please do keep in mind that this is an open-source project, that is, a collaboration, and the best way to obtain a large feature is to show up with a PR; registering your displeasure alone ("very disappointing") is fairly unlikely to be effective.

[TheQuantumPhysicist]

@TheQuantumPhysicist we're well aware! No one in the project has really made a decision not to support 32-bit targets. The reality is rather that it takes a good amount of effort to add a new compiler target (i.e., write a backend), on the order of a few months of full-time work, and none of us can afford that time away from other tasks that we also have to do at the moment.

Please do keep in mind that this is an open-source project, that is, a collaboration, and the best way to obtain a large feature is to show up with a PR; registering your displeasure alone ("very disappointing") is fairly unlikely to be effective.

Apologies for sounding self-centered and entitled in that comment. I didn't mean that I'm entitled to anything. It's just a frustrating experience I went through that complicated everything for some work I'm doing. Everyone's efforts here are highly appreciated and thank you for the great work you've been doing.

Perhaps it's just my ignorance, but I imagined that adding a 32-bit target wouldn't be that difficult, especially that wasm is 32-bit itself and that aarch64 is already supported. But I have to say that I have zero knowledge about how the backend operates, so I'm probably wrong.