Library Design

[acheong08]

I need some advice on how the websocket connection should work in JS and how data should be returned from the library. The current implementation is an experiment and should not be taken seriously.

The design in my head:
Connect(vaultID: string, keyHash: string, token: string) - Returns some sort of channel (Or whatever the equivalent is in JavaScript). Spawn a thread in the plugin to listen to push operations from the server.

Pull(uid: string) - Sends a pull request through the websocket. Returns the binary and metadata

Push(...) etc

At least this is how I would do this in Golang. However, in JS, websockets are implemented with an onmessage callback. I do not know of a way to receive objects in order and know what type of message is being received.

CC @oskardotglobal

• The path forward obi-sync#29

[acheong08]

There is of course the option to rewrite the API of the server which is trivial as the current implementation is very modular. We don't even have to break compatibility with the official API - Just extending it to make our own use easier while exposing the old endpoints.

(I would prefer to have the websocket only be used for pushes from the server and have everything else be handled through a REST API. - 1 way communication. I understand there will be some overhead with HTTP/HTTPS but I don't think that matters too much)

[acheong08]

Note

I have deleted the contents of this repository as it wasn't well thought out. This will take a few weeks (max 1 month) to get functional as I am busy relocating to a new country.

[oskardotglobal]

I'd say websockets are the best bet here.

However, in JavaScript a websocket does either text or binary data, mixing them is a bit of a pain. In this case I'd go with encoding all binary data as Data URLs (which isn't ideal, but only starts becoming a problem once we have files larger than ~256MB), so then we can send metadata and the file itself via one message.
We could also have a type field which then determines the type when deserializing JSON.
I don't know how you would order the files though, but I don't see how that's necessary either. If that's a problem, we could send a timestamp along with the metadata though

[oskardotglobal]

We also have to keep speed in mind
I am not sure on how to best incorporate this; I'll be digging through the current sync server to get a better feel for the architecture though

[acheong08]

My confusion revolves around how to pull and push data. Here is the code for pulling a file:

case "pull":
	var pull struct {
		UID any `json:"uid" binding:"required"`
	}
	err = json.Unmarshal(msg, &pull)
	if err != nil {
		ws.WriteJSON(gin.H{"error": err.Error()})
		return
	}
	var uid int = utilities.ToInt(pull.UID)
	file, err := vaultfiles.GetFile(uid)
	if err != nil {
		ws.WriteJSON(gin.H{"error": err.Error()})
		return
	}
	var pieces int8 = 0
	if file.Size != 0 {
		pieces = 1
	}
	ws.WriteJSON(gin.H{
		"hash": file.Hash, "size": file.Size, "pieces": pieces,
	})
	if file.Size != 0 {
		ws.WriteMessage(websocket.BinaryMessage, file.Data)
	}

When you send a pull request, it first sends the metadata: "hash": file.Hash, "size": file.Size, "pieces": pieces,. You must then check how many pieces there are to decide whether the next piece of data is binary. I can't wrap my head around how to do that when all incoming messages go through the onmessage callback. The metadata and binary data do not contain a tag to identify them as such. We have to somehow know what to do with the incoming data without knowing what triggered it.

One solution is to have a persistent variable that keeps track of what should come next but because it's async, it would be problematic if you receive a push signal from the server while expecting a pulled binary.

Another issue is how to return data from functions when the onmessage callback can only be set once. For example, calling Pull(UID: string) will send a pull request to the server but the function that receives the response is in whatever initiated the onmessage.

There is probably a smarter way to do this but I can't think of it

[oskardotglobal]

I feel like a more advanced websocket library like https://feathersjs.com/ or https://socket.io/ would be of good use here, which allow emitting multiple "events" on one socket which then can be handled separately, although we would have to check how that's compatible with gin & the ws library you use (I didn't try feathersjs, but I did try socketio and it's pretty good).

The problem with vanilla websockets is that it's unnecessarily hard to check if the received data is binary or json (text), so it is of no use here. By the way, if we do use socketio we may be able to mount the socket variable outside of the connection's callback scope like:

let socket; // we could also then use a maybe type or something for this
io.on("connection", (s) => socket = s);

[acheong08]

After trying out a few libraries, I still went with vanilla websockets.

My idea goes something like this:
Connect handles the initial connection. onpush(callback: Function) can be used for when the server sends sync events
pull(uid:int) sends the pull request and waits for an event to be emitted (happens when either pull response is received with 0 pieces or binary data is received.) I'm still figuring out EventEmitter

[acheong08]

https://github.com/sindresorhus/p-event works well with emitters and allow me to simply wait for data to be emitted from onmessage callback in other functions

[acheong08]

@oskardotglobal
TypeScript and Node is driving me insane.

Could you review my https://github.com/acheong08/obi-sync-lib/blob/main/tsconfig.json and https://github.com/acheong08/obi-sync-lib/blob/main/package.json?

Running tsc gives me broken imports in JS

Error [ERR_UNSUPPORTED_DIR_IMPORT]: Directory import '/var/home/acheong/Projects/obi-sync-lib/lib/src' is not supported resolving ES modules imported from /var/home/acheong/Projects/obi-sync-lib/lib/tests/vault_test.js

I can fix it manually in the JavaScript but is there a way to make sure it imports the right files on build?

[acheong08]

For example:

import { MakeKeyHash } from "./crypt";

is supposed to be

import { MakeKeyHash } from "./crypt.js";

[acheong08]

I ended up using esbuild to bundle everything together. Not sure if that is optimal but at least it works

[oskardotglobal]

You should really try https://bun.sh. It's a runtime, package manager and bundler fully compatible with node.
I'll review the whole code later today and might open a PR or two.

I don't know if the Eslint solution is ideal, I'll look into it.

[acheong08]

You should really try https://bun.sh/. It's a runtime, package manager and bundler fully compatible with node.

Just tried out Bun from their quickstart page. It worked on first try!

[acheong08]

Looks like there are still a few incompatibilities: e.g. crypto.subtle is missing when running bun build but works when running directly from the typescript