Not being able to pickle Llamacpp Embedding

[adriacabeza]

👋🏾 I tried using the new Llamacpp LLM and Embedding classes (awesome work!) and I've noticed that any vector store created with it cannot be saved as a pickle as there are cytpe objects containing pointers.

The reason is because the Vector Store has the field embedding_function which is the Callable to
the llama.cpp embedding wrapper and that contains pointers that cannot be pickled. Reproducible code:

from langchain.embeddings import LlamaCppEmbeddings
from langchain.vectorstores import FAISS
import pickle 

llama = LlamaCppEmbeddings(model_path=MODEL_PATH)
store = FAISS.from_texts(
        ["test"],
        llama,
    )
faiss.write_index(store.index, "docs.index")
store.index = None
with open("faiss_store.pkl", "wb") as f:
    pickle.dump(store, f)

which raises a ValueError:

ValueError: ctypes objects containing pointers cannot be pickled

In the case that the save_local method from the FAISS vector store is used, the index cannot be stored due to the same serialisation issue:

RuntimeError: Error in void faiss::write_index(const faiss::Index *, faiss::IOWriter *) at /Users/runner/work/faiss-wheels/faiss-wheels/faiss/faiss/impl/index_write.cpp:822: don't know how to serialize this type of index

I also tried with other libraries like cloudpickle or dill and none worked so I believe that the solution here it to convert the ctypes objects to a format that can be serialised. Do you have any recommended way to store it?

[abetlen]

Hey adri (@adriacabeza) thank you for reporting this, I've created an issue to track this in the llama-cpp-python library. It should be possible by implementing the some custom pickling methods on the Llama class. Luckily this shouldn't require any changes to be merged into langchain.

[abetlen]

adri (@adriacabeza) just pushed a fix, you should just be able to pip install --upgrade llama-cpp-python. Basically all of the underlying Llama parameters are saved to the pickle and the class is re-instantiated when you unpickle. This however implies that any changes to the Llama class (say version mismatches) or changes to things like the model_path on your filesystem are likely to break the unpickling process, just a heads up.

[adriacabeza]

Closing this as it was already fixed