Make `wasmtime-cranelift::Builder` public

[clearloop]

Feature

wasmtime-cranelift only has 1 public function builder() which is really strange...

https://docs.rs/wasmtime-cranelift/latest/wasmtime_cranelift

at least make wasmtime-cranelift::Builder public

Benefit

developers can use customized Isa to work with wasmtime-cranelift

Implementation

based on https://github.com/bytecodealliance/wasmtime/blob/main/crates/cranelift/src/builder.rs#L17

[cfallin]

Can you say more what you're wanting to do with this API?

wasmtime-cranelift is an internal crate of Wasmtime -- the engine is split into several crates for pragmatic reasons (build time, etc) but the APIs between these crates are not meant to be used by anyone else. Users of Wasmtime should use the wasmtime crate's API, and users of bare Cranelift should use cranelift-codegen, cranelift-frontend or cranelift-wasm as appropriate.

[clearloop]

Can you say more what you're wanting to do with this API?

I'm writing a project compiling wasm into EVM bytecode, wanna the 100% features of cranelift, however, the TargetIsa is hardcoded in wasmtime-cranelift that I can not integrate the translated cranelift IR with my customized Isa ))

I'm using the winch API instead now but it would be best if I can use the API provided in wasmtime-cranelift

[cfallin]

OK, it's still not totally clear to me how the pieces fit together there. You are compiling Wasm to some target: does that mean that you have ported all of Wasmtime? Or instead are you mostly using the Cranelift pipeline (is this what you mean by "wanna the 100% features of cranelift"?) and getting at it via Wasmtime APIs? What is "EVM bytecode"? Have you ported Cranelift to this as well? (Out-of-tree Cranelift backends are another somewhat unsupported case, though we've had discussions about this one before at least.)

The standard and supported way to use Cranelift to compile Wasm is to use cranelift-wasm, and plug in your environment's specific lowerings for things like heap accesses via the relevant traits. We don't support taking internal pieces of Wasmtime and invoking them in nonstandard ways, and we don't want there to be external dependencies on these aspects of our design, so I would say the private APIs are private for a reason. We'd be happy to try to help you reuse things in some other way though, if you can fill in some more detail.

[clearloop]

We'd be happy to try to help you reuse things in some other way though, if you can fill in some more detail.

Thank you for your support! just started to learn cranelift yesterday and feeling it is really helpful for expanding the usages of WASM!

The standard and supported way to use Cranelift to compile Wasm is to use cranelift-wasm, and plug in your environment's specific lowerings for things like heap accesses via the relevant traits.

hmmm, as I learnt from the code, wasmtime-cranelift does a job translating wasm to cranelift IR with optimization ( plz point me out if I'm mistaken ), while my customized target Isa is not supported by wasmtime-cranelift, I don't want to do the job wasm -> cranelift IR again on my own bcz it is risky so I want to reuse the work in wasmtime-cranelift

I want to translate wasm to the binary code of a virtual machine, if it is possible to use cranelift to make it high quality and efficient?

From the docs of cranelift-wasm, it says:

Performs translation from a wasm module in binary format to the in-memory form of Cranelift IR

I'm confused why it just does the in-memory form job but not to cranelift IR directly? From my intuition, cranelift-wasm should include the job translating wasm to cranelift IR, and for wasmtime-cranelift, it should not do the job translating wasm to cranelift IR but just operating the IR with specific purposes

---

does that mean that you have ported all of Wasmtime?

nope, I just need the part wasm -> cranelift IR with optimization

Or instead are you mostly using the Cranelift pipeline (is this what you mean by "wanna the 100% features of cranelift"?)

for example, the stack limit check in wasmtime-cranelift

and getting at it via Wasmtime APIs?

nope, I don't need wasmtime API and don't need to run the wasm in wasmtime or any other wasm runtimes

What is "EVM bytecode"?

Just the binary format for a virtual machine

Have you ported Cranelift to this as well? (Out-of-tree Cranelift backends are another somewhat unsupported case, though we've had discussions about this one before at least.)

I'm using cranelift-wasm + winch-codegen now, but I'm implementing a "base-line" compiler like what winch does xd

[cfallin]

The standard and supported way to use Cranelift to compile Wasm is to use cranelift-wasm, and plug in your environment's specific lowerings for things like heap accesses via the relevant traits.

hmmm, as I learnt from the code, wasmtime-cranelift does a job translating wasm to cranelift IR with optimization ( plz point me out if I'm mistaken ) because my customized target Isa is not supported by wasmtime-cranelift, I don't want to do the job wasm -> cranelift again on my own bcz it is risky that's why I want to reuse wasmtime-cranelift

OK, I see the confusion here: wasmtime-cranelift does not translate Wasm code to Cranelift IR (CLIF). wasmtime-cranelift is only the bridge between Wasmtime and Cranelift. If you are not using or porting Wasmtime, you don't want to use wasmtime-cranelift.

Instead, cranelift-wasm is the crate (used by wasmtime-cranelift) that actually does the Wasm to CLIF translation. It requires some "hooks" to be filled in that are specific to the Wasm runtime the code will be running under.

I want to translate wasm to the binary code of a virtual machine, if it is possible to use cranelift to make it high quality and efficient?

Maybe; it depends what sort of translation you want. Cranelift is an optimizing compiler, so if you want the Wasm -> custom bytecode translation to do optimizations, you'll for sure get that. However it also makes a bunch of assumptions about the target architecture: it's really meant for "real hardware" ISAs like x86 or aarch64, and porting it to emit a custom bytecode may or may not be easy, depending on the bytecode's design. For example, Cranelift uses a register allocator and assumes that the target ISA has a fixed number of physical registers. If your bytecode is stack-based rather than register-based, that may be a problem.

Porting Cranelift is also not easy (no compiler is really super-easy to retarget; "real ISAs" tend to be large). The initial PR to add support for RISC-V, our latest addition (#4271), was 21k lines of code.

does that mean that you have ported all of Wasmtime?

nope, I just need the part wasm -> cranelift IR with optimization

Or instead are you mostly using the Cranelift pipeline (is this what you mean by "wanna the 100% features of cranelift"?)

for example, the stack limit check in wasmtime-cranelift

and getting at it via Wasmtime APIs?

nope, I don't need wasmtime API and don't need to run the wasm in wasmtime or any other wasm runtimes

OK, yeah, given the above, using wasmtime-cranelift is definitely not the right answer. In fact the stack-limit check you link is closely related to Wasmtime's data structures (its VMRuntimeLimits) so if you're not actually using Wasmtime, it doesn't make any sense to use that implementation. You'll need to develop your own CLIF lowerings for Wasm features that target your own bytecode VM.

I'll make a note on terminology here too: you need some "runtime" for the Wasm specifically, on top of your custom bytecode VM's runtime. In this context, "runtime" means support for Wasm semantics and state: for example, handling Wasm memories and tables. It could be a very simple mapping, but somehow you need to decide what a "memory load" or "table set" op becomes, and this is what you provide in your "hooks" to cranelift-wasm. wasmtime-cranelift provides code for all of this that assumes Wasmtime's data structures for memories, tables, function references, etc. exist. You'll need equivalents for all of these in order to run Wasm.

What is "EVM bytecode"?

Just the binary format for a virtual machine

Have you ported Cranelift to this as well? (Out-of-tree Cranelift backends are another somewhat unsupported case, though we've had discussions about this one before at least.)

I'm using cranelift-wasm + winch-codegen now, but I'm implementing a "base-line" compiler like what winch does xd

I'm not sure I understand what you're doing here: cranelift-wasm produces CLIF, an input format specific to Cranelift, but winch-codegen consumes Wasm and directly produces machine code, and has no understanding of CLIF at all. How are you combining the two?

[clearloop]

Cranelift is an optimizing compiler, so if you want the Wasm -> custom bytecode translation to do optimizations, you'll for sure get that.

yeah, this is what I want! and the compiler time doesn't matter to my project!

If your bytecode is stack-based rather than register-based, that may be a problem.

Sadly it's stack-based...but while my target vm's instructions are super easy, I'll find it out if it is possible and necessary soon!

In fact the stack-limit check you link is closely related to Wasmtime's data structures (its VMRuntimeLimits) so if you're not actually using Wasmtime, it doesn't make any sense to use that implementation.

got you now, so what I can do is learning from the implementation of wasmtime-cranelift to write my own CLIF

but winch-codegen consumes Wasm and directly produces machine code

I just implemented the TargetIsa trait from winch-codegen for using the private CodeGen implementation xD

thank you for your patient explanation @cfallin, I'm closing this issue since ur right and now I know what to do next!

[clearloop]

@cfallin ur totally correct xD, I have to write my compiler on my own now since my target vm is stack-based... but wasmparser and the compiler ( both winch and cranelift ) implementation have inspired me really a lot! thank you for your great work!

btw if there is any public methods in cranelift that I can re-use for building an optimizing compiler for wasm like cranelift but to stack-based vm target implementation, I'd be really grateful if you can point me out ; )