1
$\begingroup$

I have a question regarding the pre-training section (in particular, the Masked Language Model).

In the example Let's stick to improvisation in this skit, by masking the word improvisation, after applying BERT (followed by the FFNN and Softmax), by looking at the probabilities of each of the possible English words, we are able to correctly predict that the masked word was improvisation.

Is it possible to actually play with this by using my own examples? I was wondering if I can input a sentence in a different language (from the multilingual model) and have a sorted list of the most probable words that were masked in the original sentence. If it's possible, what needs to be tweaked?

Any help would be greatly appreciated.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

pip install transformers

Then try this 

from transformers import pipeline
nlp = pipeline("fill-mask", model="bert-base")
nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thank you, that works! Just one mention: if anyone tries this and gets the message: Exception: Impossible to guess which tokenizer to use. Please provided a PretrainedTokenizer class or a path/identifier to a pretrained tokenizer. they should specify the model in the pipeline (e.g. "bert-base-cased") $\endgroup$
    – moz_szt
    Apr 17, 2020 at 9:56
0
$\begingroup$

You can use model='bert-base-uncased' with transformers.pipeline:

pip install transformers


from transformers import pipeline
unmasker = pipeline('fill-mask', model='bert-base-uncased')
unmasker("a [MASK] in the watergate saga")

This works for me on Google Colab.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.