Advanced HTTP events

[xpepermint]

The documentation describes HTTP related events pretty well. I try to understand advanced use cases which are not covered by the docs.

I'd like to standardize all incoming requests of my SAAS service by forcing cloud events everywhere. This would include API calls performed by users over the HTTP protocol (e.g. REST, RPC). The spec in this repo is pretty forward. Based on HTTP Protocol Binding nothing prevents such use case. If I'm correct, then a user would send event metadata as HTTP headers and use application/json content type for GET/POST requests and application/octet-stream for binary stuff like file uploads. The only restriction I see is the size limit for intermediaries and consumers which is expected to be up to 64 KB - this blocks file upload implementation. By ignoring this limit the HTTP event received from a user would look like this:

const httpEvent = {
    "specversion" : "1.0",
    "type" : "com.myservice.api.files.upload",
    "source" : "https://myservice.com/api/files",
    "subject" : "upload",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "datacontenttype" : "application/octet-stream",
    "data" : HTTPRequestObject // stream instance (`req`)
};

I wonder why Google's implementation of HTTP cloud function doesn't provide the event object. It seems like Knative also follows Google's pattern. AWS, however, agrees that a user sends an event object where they attach custom properties.

// Google
exports.helloHttp = (req, res) => {};
exports.helloEvent = (event, context) => {};
// AWS
exports.helloHttp = async (event) => {};

If we ignore how cloud functions work today, cloud events could also start HTTP event streaming (SSE) and there's a bunch of other use cases. At the moment it seems to me that cloud events are only meant for small background triggers and not as an overall event standard. Am I correct? I remember a talk almost a year ago where a guy said that an event carries only data for a consumer to create another request to access actual data. Maybe this is still true.

I would appreciate comments on what I wrote above and some more words about the conceptional-level overview of how the cloud events should be used. It would be great for this spec to be more opinionated to eliminate confusion. Thank you.

[duglin]

The only restriction I see is the size limit for intermediaries and consumers which is expected to be up to 64 KB - this blocks file upload implementation

Actually, it allows for larger sizes but in terms of interop we only require up to 64KB. So your usecase can work just fine as long as each hop along the message path can support the larger sizes.

"subject" : "upload",

I think this would be something more specific to the entity uploaded or the entity into which the thing was uploaded (like a bucket). A generic word like "upload" doesn't seem quite right.

event objects

I can't speak for other cloud providers, but I believe that Knative does pass along the CloudEvents metadata since it'll be part of the message sent to the knService. It just might not be encapsulated as an "event object" but rather as http headers. @n3wscott can double-check me on this though.

At the moment it seems to me that cloud events are only meant for small background triggers and not as an overall event standard.

This might get into a bit of a philosophical discussion, but to me you can use CloudEvents for any size event (or message). Since CloudEvents doesn't really get into the transport aspect of messages, it's just adding a few bits of metadata, it should be used any place you're looking for "normalization" of eventing-like payloads, and I don't see why size or "background" aspects would impact that decision.

I remember a talk almost a year ago where a guy said that an event carries only data for a consumer to create another request to access actual data.

Perhaps this was in reference to our discussion around supporting "claim check" patterns, where we didn't want large payloads in the event rather we want just a pointer to it. And while I do agree with this sentiment, I also don't view it as a rule that everyone MUST adhere to. I think there are many usecases where for performance or access control reason you don't want to bloat your events too much and want to keep them small. But, I think there will be other cases where people really do want/need the large payload to be part of the event, and that's ok - as long as the transports/hops can deal with it. Remember, a lot of the guidance we put into the spec/primer was designed to increase interop across as many different environments as possible, but if you're in an environment where all parties agree to a different set of constraints then that's fine.

[xpepermint]

Cool. @duglin your answers are very helpful so thank you for that. I really like this spec and I had to open this discussion because CloudEvents is a specification, a protocol, and I see protocols as a group of 0 and 1 - so no gray areas, no chance for 0.2, 0.5 :). People who implement such protocols must have answers on all the edge cases.

Based on what you wrote I would suggest updating the documentation with these notes. The size should not be explained as a restriction but a best practice approach for X Y cases. I also like the explanation around the normalization of payloads.

I'll wait for @n3wscott to add his thoughts, then we can close this issue.

[n3wscott]

I think I have some perspective to help here. I think it could be an anti-pattern to talk about cloudevents and the protocol at the same time. I have spent a lot of time trying to decouple how the event gets to the consumer and what the event looks like to the consumer in the go sdk for cloudevents. As a result, the consumer function has the same signature when consuming events from AMQP, NATS, HTTP Binary, and HTTP Structured, etc.

Take a look at https://github.com/n3wscott/sockeye for an example of CloudEvents consumed over a websocket. Not officially in the spec, but I leverage the structured encoding in the spec.

Knative is based on http requests. We are parsing the incoming request into a cloudevent and then binding those to other transports if you want, like AMQP, NATS, Pub/Sub, etc. The bits inside of knative are connected via HTTP as a standard for now until we can get delegation or negotiation between the components.

You could implement a FaaS or SaaS ontop of Knative and expose only the event object, Knative makes no opinions on this.

Perhaps the answer to the question about both cloud function and knative is that both are not coupled to cloudevents. But cloudevents works if you use a library to help convert the raw input to the function to the cloudevent object.