Tracking issues of OpenDAL wasm32-unknown-unknown target support

[Xuanwo]

WebAssembly (WASM) is a modern binary instruction format for a stack-based virtual machine. It is designed as a portable compilation target for high-level languages like C/C++ and Rust, enabling deployment on the web for client and server applications. WASM aims to enable near-native performance for web applications, providing a more efficient and secure alternative to traditional JavaScript. Its key features include fast loading times, efficient execution, and the ability to run alongside JavaScript, allowing developers to build complex client-side web applications that perform better than those written solely in JavaScript.

By adding WASM support for opendal, we will allow users to use opendal in targets like wasm32-unknown-unknown. In a short word, users can use opendal in web browser!

We have addressed most blockers by #3796 and adding a basic test for s3 #3802. It's time for us to move forward.

This tracking issue is used to track our current progress of wasm support. Welcome to join in us!

Tasks

Add OPFS support

After opendal is ready for WASM, we can start to work for opfs now!

• Add Origin Private File System support #2442

Add support for more services

Please run the following command to check if opendal can build under wasm32 arch:

cargo build --target wasm32-unknown-unknown --no-default-features --features=services-s3

Visit #3796 for possible fix or ask directly in this issue.

It's possible that some services can't support wasm at all, like fs. Please also provide your feedback here. We will collect them into a seperate doc.

• alluxio
• atomicserver
• azblob in feat(services/azblob): available under wasm32 arch #3806
• azdls
• azfile
• b2
• cacache (can't support)
• cloudflare_kv
• cos
• d1
• dashmap
• dbfs
• dropbox
• etcd (can't support)
• foundationdb (can't support)
• fs (can't support)
• ftp
• gcs: feat(core): Make gcs available on wasm32 arch #3816
• gdrive in feat: make services-gdrive compile for wasm target #3808
• ghac
• gridfs
• hdfs (can't support)
• http
• huggingface
• ipfs
• ipmfs
• libsql
• memcached
• memory
• mini_moka
• moka
• mongodb (can't support)
• mysql (can't support)
• obs
• onedrive
• oss
• persy (can't support)
• postgresql (can't support)
• redb (can't support)
• redis (can't support)
• rocksdb (can't support)
• s3 (supported in feat: Add edge test s3_read_on_wasm #3802)
• seafile
• sftp (can't support)
• sled (can't support)
• sqlite (can't support)
• supabase
• swift
• tikv (can't support)
• upyun
• vercel_artifacts
• webdav
• webhdfs

Testing

• Integrate behavior test for wasm target

Benchmark

• Add benchmarks for opendal wasm

Existing issues

• wasm-pack test --node doesn't work for s3_read_on_wasm
• wasm-pack test --chrome --headless doesn't work for s3_read_on_wasm #3809

[Zheaoli]

FS style target can supported by wasm32-wasi.

I will take handle of this

[Xuanwo]

FS style target can supported by wasm32-wasi.

I will take handle of this

You can start a new issue for this. It's out of the goal of this tracking issue.

[Zheaoli]

For now, I think the service based on the TCP socket is not working because the wasm32-unknown-unknown is not support TCP yet, the wasm32-unknown-emscripten can support TCP socket but need some hackc on Server

The TCP service is some service like following bellow:

1. MySQL
2. PostgreSQL
3. MongoDB
4. Redis

[Zheaoli]

Can we be used on wasmtime?

Can we be used on wasmedge?

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.

The WASI runtime is not support for the wasm-bindgen which is dependent by reqwest now.

If we want to support the wasi runtime, we may need the other HTTP client which is compatible with the WASI spec

[Xuanwo]

For now, we can't use it on both wasm runtimes, because both of the runtimes is based on wasm32-wasi I have reminded it in the previous comment.

Thanks, let's remove them. I just found that support wasm32-unknown-unkown is totally different from support wasm32-wasi. Lesson learnt.

[Zheaoli]

For now, the memory service can both on wasi(means can run in wasmtime and wasmedge, tested) and wasm32. The fs services need remove the tokio::fs

[Zheaoli]

For the record,

wasm-pack test --chrome --headless and wasm-pack test --chrome is both working.

But need some special S3 settings like

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "POST",
            "PUT",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 3000
    }
]

[suyanhanx]

incompatible crate found: ring@0.17.5
When I tried to compile with services-gcs, error reported:
No available targets are compatible with triple "wasm32-unknown-unknown"

cargo tree -i ring
ring v0.17.5
├── jsonwebtoken v9.2.0
│   └── reqsign v0.14.6
│       └── opendal v0.43.0 
├── rustls v0.21.9
│   ├── hyper-rustls v0.24.2
│   │   └── reqwest v0.11.22
│   │       ├── opendal v0.43.0 
│   │       └── reqsign v0.14.6 (*)
│   ├── reqwest v0.11.22 (*)
│   └── tokio-rustls v0.24.1
│       ├── hyper-rustls v0.24.2 (*)
│       └── reqwest v0.11.22 (*)
├── rustls-webpki v0.101.7
│   └── rustls v0.21.9 (*)
└── sct v0.7.1
    └── rustls v0.21.9 (*)

Related issue: briansmith/ring#657
It looks like ring isn't going to support wasm for a while.

[Xuanwo]

When I tried to compile with services-gcs, error reported:

Let's mark gcs as unsupported since gcs will require jsonwebtoken (and ring) for JWT.

[gabrielgrant]

@suyanhanx it seems surprising that ring would be failing in that way: the issue you linked seemed to say that it should be able to build for wasm32-unknown-unknown: briansmith/ring#657 (comment)

the author (@briansmith) does note that some functionality isn't available yet, but is open to extending support. do we know what is being used that isn't supported?