Unification of /processes and /process_graphs

[soxofaan]

This is follow up issue of #305/#309, which added a namespace property to process graph nodes to identify whether the process_id property refers to a predefined process or a user defined process.

Possible next steps:

• merge GET /processes and GET /process_graphs and work with an additional namespace request parameter to select PDP or UDP (or even other namespaces)
• unify all /processes and /process_graphs requests to something like /processes/{namespace}/{process_graph_id}

[jdries]

This issue is becoming relevant again for us: we'd like to allow users to expose their process graphs. This is supported by the namespacing, but it seems they are not yet discoverable?
I would try to avoid breaking the API for this, at least not in a 1.1 version. So certainly the /processes endpoint should not change in an incompatible way.

So I'm proposing something like:
/namespaces -> list all namespaces
/namespaces/{namespace} -> list all processes in a namespace
/namespaces/{namespace}/{process_graph_id} -> list process metadata
This resembles the GitHub approach: our namespaces would be called an 'organization' in GitHub. Users could store process graphs in multiple organizations (namespaces).

By not reusing /processes or / process_graphs we are fully backwards compatible. I'm not yet sure if 'namespaces' is the best name?

[jdries]

Rethinking this: the use case of sharing user defined processes is also what the 'marketplace' in openEO platform is supposed to do.
So we could also consider defining an entirely separate microservice, that can move at its own pace, so we can move at a faster pace compared to openEO versions? (Meaning in the beginning, when we are defining and changing such a service.)

[aljacob]

What do you mean @jdries with a seperate microservice? Just an arbitrary http service that is able to work with openEO processes or process_graph and store and offer them? I think the namespaces proposal from you, from yesterday was not bad. We can still keep that as an experimental endpoint for now, without a new API release.

Thinking more about it...
I have to say I am also a bit confused by the duality of /processes and /process_graphs now that I have implemented version 1. As process_graphs are anyways always wrapped in a process when send to the back-end for processing, maybe it would actually make sense, to organize also their storage and sharing via processes. Similar to what @soxofaan has proposed originally.

[m-mohr]

The /process_graphs endpoint is a legacy from the old 0.4/... days, where this was the endpoint for process graphs. The name currently is also wrong, because it serves user processes, not process graphs only. As the change from process graphs to processes and namespaces came in rather late in 1.0, we didn't really made the big change to completely refactor the endpoints to avoid too many changes. Ultimately, the desire would be to have everything under /processes.

I'm not a big fan of the /namespaces approach, I must admit. This needs quite a lot of round trips to get the actual process information if it's the only way to get the information. Also, namespace is less clear than processes. I like the proposal from @soxofaan. This can likely be done with relatively low work and /process_graphs could just redirect for some time. (Now I'm happy that we never did /processes/:id as this now allows us to use the namespace under /processes as we like).

This would be somewhat backward-compatible, I think:
/processes - pre-defined processes, stays as it is, could be an alias for /processes/backend
/processes/:namespace - all processes under the namespace, could be a user id. Shortcut for the user id could be /processes/user
/processes/:namespace/:processid - GET/DELETE/PUT for individual processes

Redirects:
/process_graphs => /processes/:userid
/process_graphs/:id => /processes/:userid/:id

The only thing missing is a list of namespaces, but that could be retrieved via GET / either as part of the endpoint list or as a separate property.

This way we could have it in a 1.1 release, I think. It could also just be an API extension for now.
At the same time, we should think about sharing though.

A microservice at the EOSC marketplace or the Hub could just implement the same API. We had that earlier for 0.4 at the Hub, but had no time yet to update to 1.0.

[jdries]

The proposal to allow /processes/:namespace and /processes/:namespace/:processid is also fine for me.
Would the fact that '/processes/somename' can be both a namespace or a process be problematic? Like in the defining the typing of such a call?

I also created a diagram of a broader marketplace design, to give an idea of what I'm thinking of in terms of additional use cases that I'd like to support:

[m-mohr]

Would the fact that '/processes/somename' can be both a namespace or a process be problematic? Like in the defining the typing of such a call?

/process is the only endpoint at the moment - /processes/:processid is not defined in the API and would not be in the proposal! It's only allowing /processes/:namespace. See my comment above.

I also created a diagram of a broader marketplace design

Looks interesting, but is also quite an effort. How's the relationship with the Hub?