Possibility of reducing KVCache size

[yuanfz98]

Hello,

I wonder if it is possible to reduce the KVCache size by reducing the dimension cfg.max_seq_len.

[markian-rybchuk]

Yes, reducing max_seq_len (and decode/prefill batch size) would reduce KVCache size