Accept Encoding has different semantics than HTTP standard

[Zaczero]

connectrpc/connect-python#96

Today I learned, that connectrpc uses the standard Accept-Encoding header in a non-standard way: ordered list. This causes inefficiencies with existing software, like Google Chrome, that for historical compatibility reasons start with "gzip, deflate" even though zstd is clearly more optimal for dynamic responses like in the connectrpc case. Accept-Encoding like "gzip, deflate, br, zstd" means that the server should pick whatever algorithm without client preference. Changing the standard semantics seems like the wrong way to go about it. I believe the connectrpc should either properly handle the q= value semantics or use a new header to not conflict with existing software. Otherwise I end up in a situation where I create a service that is accessible via HTTP and most browser users have higher response latency and bigger response size for no real reason.

[anuraaga]

For context, I researched how some servers work, and found they all seem to behave similarly in that they use the order compressions are registered on the server, rather than the client's list, though Envoy has an explicit option to override ordering to a specific one in the list. FWIW, I'm happy to add the ability to define the preference on the server side in connect-python as an option, even if it isn't in the spec, if there is at least some consensus here among maintainers that it seems like a reasonable thing to support.

/cc @stefanvanburen @jhump

[timostamm]

RFC 9110:

If no "q" parameter is present, the default weight is 1.

I'm not aware that HTTP specifies how the server should choose from a list of equal weights. Connect builds on HTTP and does specify how the server should choose, but that doesn't necessarily mean that it is at odds with HTTP semantics.

That said, all major browsers today still send Accept-Encoding: gzip, deflate, br, zstd (Chrome, FF) or gzip, deflate, br (Safari), and it's a bummer that Connect servers are bound to always choose gzip.

The relevant passage in the Connect Protocol Reference, under the section "Unary-Reqest":

Servers should treat Accept-Encoding as an ordered list, with the client's most preferred encoding first and least preferred encoding last. This is a simplification fo standard HTTP semantics that excludes quality values.

Unfortunately, I do not know whether the motivation was just to keep implementations simple, or if there were other considerations, but amending the protocol is a possibility if maintainers agree.

[anuraaga]

I have been thinking about the compatibility of changing existing behavior and feel

• Most connect implementations require opting in to zstd / br. I think connect-python was unique in using dependency detection to automatically enable them

• For those implementations, clients would also need to be configured to take advantage of this by moving gzip back

So it seems hard to imagine a situation where changing to server side preference would actually have a chance to harm.

I am planning on sending a PR to tweak the spec that servers SHOULD accept an ordered list of compressions to be registered and control preference, removing the current client controlled behavior. Since it is SHOULD, if out of an abundance of caution libraries make it opt-in, it will be fine. Though I don't know how many actually have an ordered list on the server side.

If the above change were accepted, I would probably change connect-python to only use the new behavior since it's not 1.0 yet - I have already sent a change to not do automatic detection to align with other languages and make it easier to make this change.

If this sounds good, please leave a thumbs up, with at least one I'll send that PR

[Zaczero]

I think SHOULD is a good middle ground. New clients should send an ordered list to comply with existing server software, and new servers should just do whatever makes sense for their use case, without forcing a specific behavior.

[thezachdrake]

The only problem with updating the spec, even with a SHOULD caveat, is we will start to see diverging behavior in ConnectRPC implementations which is not ideal. I think it is unlikely we would merge the change to the spec until its clear if/how at least the official implementations would work. That being said @anuraaga if you want to open a PR on the spec we can move this conversation there!

[Zaczero]

even with a SHOULD caveat, is we will start to see diverging behavior

I think that's the point if we want to resolve that issue. SHOULD implies that the behavior may diverge. By not following HTTP semantics in browser contexts, users get subpar experiences. The spec was not made with browser users in consideration.

My question (I'm new) is why would one care about "at least the official implementations would work" if the updated spec would not enforce any specific behavior. I see it as they may do whatever works best for their specific implementation.

[anuraaga]

Sorry for the delay, I have gone ahead and sent #322 to discuss further if we can make this change. Personally, I would like to improve the experience for browsers if possible.

I see it as they may do whatever works best for their specific implementation.

@Zaczero In precise terms, implementations would continue to be compliant by using loose wording. But we do also want the servers to generally behave the same to reduce surprises, for example if a user is using connect-go already and introduce connect-node to their stack, they may have some expectations on how it will work. So ideally we can find a way forward that allows the languages to continue to behave as similarly as possible.