Using LLM in low memory (CPU) local environnements

Question

Kind of a generic question, but I wonder if there is any solution on using LLM in low-memory environments, that is CPU environments that can't fit the whole model in memory. I am mostly thinking about Mistral Instruct and its MoE version, Mixtral.

I am generally interested in demo code.

Answer 1

You can use the LLM implementation of llama.cpp together with the Q4_K_M quatization of Mixtral.

This is reported to work on just CPU with not even 32Gb of RAM. Other reports suggest that at least 24Gb of RAM would be needed.

Answer 2

There are some model optimisation techniques out there which help in running the model on low end hardware. Implementing some or all of them will for sure help in your case.

1. Quantization : Basically this technique compresses the memory requirements of the weights. So if your weights are fp32, you can compress them into fp16, fp8 or even int8. This reduces the memory footprint of your LLM which can help you in loading the model in your CPU RAM.

   Here is an article explaining the theory behind it.

   Here is a github repo for the Mixtral model where they have implemented Quantisation as well as an efficient model loading startegy!

   Here is an implementation of quantisation being done on GPT-2 LLM.

   Here is another implementation of quantisation where Microsoft's ORCA model is being Quantised and LangChain is also used.

2. Pruning : Pruning is when you cut the unnecessary weights of the model. The aim is to get a computationally cost-efficient model that takes a less amount of time in training. The necessity of pruning on one hand is that it saves time and resources while on the other hand is essential for the execution of the model in low-end devices such as mobile and other edge devices.

   Here is an article explaining the theory behind pruning and different types of pruning.

   Here is a Github repo which will get you started with LLM Pruning. They have pruning options for multiple LLM's.

   Wanda is another type of pruning that can be used to overcome some of the limitations of conventional pruning!

This comprehensive huggingface article also lists a couple other optimisation techniques that can be used.

Let me know if it works for you.

Cheers!