0
$\begingroup$

I have a pretrained Llama-2 model in the models_hf directory and two fine-tuned adapters: a summarization adapter in ./tmp/llama-output and a chat adapter in ./tmp/chat_adapter. The details of the code are in another question.

For inference, I'd like to use all three components: the pretrained model, summarization adapter, and chat adapter. However, I'm unsure about the memory requirements and the best approach.

Do I need to:

Option 1: Load one instance of the pretrained model (e.g., 10GB GPU memory), and then load the two separate instances of adapters (e.g., 100MB each), resulting in a total memory usage of 10GB + 200MB?

OR

Option 2: Load two separate instances of the 10GB pretrained model and stack the summarization adapter on one and the chat adapter on the other, resulting in a total memory usage of 20GB + 200MB?

Additionally, could you provide a code example or steps on how to load and use these components for inference effectively? I'm looking for guidance on memory management and loading processes to ensure smooth and efficient inference with this setup.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Someone appears to have fixed this problem, just replace the image models with language models. But looking forward to better solutions.

https://github.com/huggingface/peft/blob/main/examples/lora_dreambooth/lora_dreambooth_inference.ipynb

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.