I have a pretrained Llama-2 model in the
models_hf directory and two fine-tuned adapters: a summarization adapter in
./tmp/llama-output and a chat adapter in
./tmp/chat_adapter. The details of the code are in another question.
For inference, I'd like to use all three components: the pretrained model, summarization adapter, and chat adapter. However, I'm unsure about the memory requirements and the best approach.
Do I need to:
Option 1: Load one instance of the pretrained model (e.g., 10GB GPU memory), and then load the two separate instances of adapters (e.g., 100MB each), resulting in a total memory usage of 10GB + 200MB?
Option 2: Load two separate instances of the 10GB pretrained model and stack the summarization adapter on one and the chat adapter on the other, resulting in a total memory usage of 20GB + 200MB?
Additionally, could you provide a code example or steps on how to load and use these components for inference effectively? I'm looking for guidance on memory management and loading processes to ensure smooth and efficient inference with this setup.