Feat request: backend agnostic compression

[tasiotas]

Hi,

Would it be possible to add an option for compressing values before they are sent to the cache backend?

I see that Badger supports zstd, but others don't (I use Redis).

thanks

[darkweak]

Hi @tasiotas, actually I'm not compressing data before send them to the storage provider, I need to have some metrics about the cost to decompress data each time we want to get them.

[tasiotas]

sorry, I wasn't clear enough.
I would like to compress data before sending to backend.

I am aware of possible costs of compression/decompression.
Perhaps giving user option to choose compression level and algorithm, so we can decide costs vs benefits of saving memory.

http://facebook.github.io/zstd/
Seems like zstd is the king 👑

Compressor name	Ratio	Compression	Decompress.
zstd 1.4.5 -1	2.884	500 MB/s	1660 MB/s
zlib 1.2.11 -1	2.743	90 MB/s	400 MB/s
brotli 1.0.7 -0	2.703	400 MB/s	450 MB/s
zstd 1.4.5 --fast=1	2.434	570 MB/s	2200 MB/s
zstd 1.4.5 --fast=3	2.312	640 MB/s	2300 MB/s
quicklz 1.5.0 -1	2.238	560 MB/s	710 MB/s
zstd 1.4.5 --fast=5	2.178	700 MB/s	2420 MB/s
lzo1x 2.10 -1	2.106	690 MB/s	820 MB/s
lz4 1.9.2	2.101	740 MB/s	4530 MB/s
lzf 3.6 -1	2.077	410 MB/s	860 MB/s
snappy 1.1.8	2.073	560 MB/s	1790 MB/s

[tasiotas]

after thinking about it for a while:

1. zstd is not the best algorithm for this case. As none of the browser can accept responses encoded with zstd, it would have to be decompressed before sending to the client. With gzip and brotli this can be avoided, so we only have to pay the cost of compressing.
2. I can encode responses with Caddy's encode directive before it's sent to cache, but is that going to work if souin receive encoded response? Seems ok at first glance, but you might have a better idea.

Might not need any implementation in souin after all!

Caddyfile

{
  debug
  order cache before rewrite
  order cache before encode
  cache {
    log_level debug
    allowed_http_verbs POST
    ttl 20000s
    api {
      souin
    }
    redis {
      url redis:6379
    }
}

website.com {
  handle /api* {	
    encode {
      gzip
    }
    reverse_proxy http://django:8000
    cache
  }
}

[darkweak]

That's great if it can work without any additional code! But I wonder if the encode respect the no-transform Cache-Control directive. That means in the request context that the requests should not be altered and the same in the response context.

[tasiotas]

unfortunately it ignores it.

I tried curl with -H "Cache-Control: no-transform" and it returned and cached gziped response

[francislavoie]

FWIW:

  order cache before rewrite
  order cache before encode

This doesn't make sense, only the last order for a directive you set will take effect. So effectively it will be before encode, always.

What order does is insert the directive at some spot in this list https://caddyserver.com/docs/caddyfile/directives#directive-order at the position you specify, so then the Caddyfile directive sorting logic can do its thing to make sure the JSON output is in the right order, so it runs in the right order.

[tasiotas]

thank you @francislavoie for pointing out my silly line of code.
I forgot to remove first line during debugging.

[tasiotas]

compression is working well, thanks!

[Cluster2a]

compression is working well, thanks!

How excatly did you accomplish this? Do you mind sharing the config? Does it has to do with the order ode wih the Cache-Control?

Is the compressed content stored in redis, or the uncompressed one? I want to reduce the memory consumption by storing compressed data into redis.

[tasiotas]

yes, in my case it was about order. make sue whatever you send to redis is compressed if you want to reduce its memory consumption.

[Cluster2a]

yes, in my case it was about order. make sue whatever you send to redis is compressed if you want to reduce its memory consumption.

That is strange... the order does not seem to impact the key size in redis... I need to check again - thanks for your input :)

Edit: I am wondering how this is supposed to work. The default encode prefers zstd and falls back to gzip (https://caddyserver.com/docs/caddyfile/directives/encode). Lets say our Client accepts zstd and the content is cached in redis compressed with zstd. A client, hitting the same route, not able to handle zstd but the cache serves zstd compressed content. Wouldn't you have to save both caches to redis (one for zstd and one for gzip)?

@darkweak, could you add some information here?