I'm using Huggingface to do inference on llama-3-B. Here is my model:

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-Instruct-bnb-4bit",
    max_seq_length = 2048,
    dtype = torch.float16,
    load_in_4bit = True,

model = FastLanguageModel.get_peft_model(
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = True,
    random_state = 42,
    use_rslora = False,
    use_dora = False,
    loftq_config = None,

My issue is that when I prompt the model, it outputs "prompt + generation" rather than just outputing the generation. Here is where I prompt it:

inputs = tokenizer(prompts[:2], return_tensors = "pt", padding=True).to("cuda")
outputs = model.generate(**inputs, max_new_tokens = 256, use_cache = True)
predictions = tokenizer.batch_decode(outputs, skip_special_tokens = True)

E.g. if the input prompt i.e. prompt[0] is "What color is grass?" the output is like "What color is grass? Green." but I want it to just be "Green.".


1 Answer 1


Given that Llama is a decoder-only model, the output of the model (i.e. outputs) is just the whole sequence. You can, nevertheless, decode only the new tokens easily like this:

predictions = tokenizer.batch_decode(
    outputs[:, inputs.shape[1]:],
    skip_special_tokens = True
  • $\begingroup$ Thank you for your answer. If you don't mind, I also got this warning when printing the output. I think I understand why, but how come it didn't cause my output to be incorrect?: Setting pad_token_id` to eos_token_id:128001 for open-end generation. A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer.` $\endgroup$ Commented Apr 26 at 19:03
  • $\begingroup$ The comments in this issue might be useful for that problem. $\endgroup$
    – noe
    Commented Apr 27 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.