I am doing a POC on LLM text generation. I have one AWS p3.8x instance which has 4 GPUs each of 16 GB size. I am pretty new to use LLM and GPU. When I am trying load one LLM pertained model (WizardLM) in GPU, it is saying 16 GB is not sufficient for this. So my question is how can I load the model using all 64 GB?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ For AWS, you can check out SageMaker - huggingface.co/docs/transformers/perf_train_gpu_many though I've never used personally, it's supposed to split the model up over multiple GPUs. $\endgroup$– brewmaster321May 19 at 15:40
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Using multiple GPUs usually means that the whole model is copied into the memory of each of them. In Pytorch this is achieved with nn.DataParallel or nn.parallel.DistributedDataParallel. This however, is not what you want.
It is possible to load parts of a model into different GPUs and distribute the computation among them. This, however, needs specific code logic to distribute and coordinate the different parts. It is not possible to automagically split a model into parts among different GPUs.
Your options are:
- Use a smaller model that fits on 16Gb.
- Use a GPU with enough memory to fit your current model.
- Use a quantized version of your model that is small enough.
- Perform CPU inference. This may be very slow. You may check if there is a C++ implementation for your model using parallelized CPU instruction sets to make inference fast; for instance, for Llama you can use llama.cpp.
