1
$\begingroup$

I am using a falcon 7B model for a chatbot without any finetuning with the following code

model_name = "ybelkada/falcon-7b-sharded-bf16"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    trust_remote_code=True
)
model.config.use_cache = False
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model=model,
    tokenizer="gpt2",
)

result = generator("Hi")
print(result)

the result isnt as expected and it outputs [{'generated_text': 'Hi8\x10=:AHi8\x10>Hi8\x10>:AHi8\x10?'}]. How can i fix this and make it output a proper response

$\endgroup$
2
  • 2
    $\begingroup$ Does it work better without quantization? $\endgroup$
    – noe
    Commented Jan 15 at 14:54
  • $\begingroup$ no there is no difference without it $\endgroup$ Commented Jan 15 at 16:35

1 Answer 1

2
$\begingroup$

You are using the GPT-2 tokenizer instead of the model's own tokenizer. Try getting the tokenizer from the model instead:

from transformers import AutoTokenizer
import transformers

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)
$\endgroup$
2
  • $\begingroup$ @SaketVempaty did my advice help make the model generate normal text? $\endgroup$
    – noe
    Commented Jan 16 at 8:11
  • 1
    $\begingroup$ yes it did although the falcon model probably needs to be fine tuned for giving meaningful answers $\endgroup$ Commented Jan 17 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.