Recreate KV cache from residual stream

[edt-xx]

Is the process advocated by Chris Hay in
https://www.youtube.com/watch?v=TYgCRPCAFhE
applicable to llama.cpp?

[x4080]

Yes I just saw the video, and you beat me on this - This will change everything for local AI

[edt-xx]

IF it can be applied generally. From the video it looks like this might be the case, but my understanding of the details it lacking hence this post.

[Nekotekina]

https://github.com/chrishayuk/chuk-lazurus/blob/main/examples/inference/gemma_rs_compiled.py
This code contains relevant information about what the video really is trying to say. RS cache is often bigger than KV cache. It just claims some "dark capabilities" which I don't understand.

[daniel-ffm]

Speed boost incoming =D the residual will save a lot of monies!
KV-direct paper
https://arxiv.org/pdf/2603.19664

[daniel-ffm]

#21097

someone has implemented it, i have run it and it works.

code would need some cleanup