New answers tagged


Here I would show how we can use transformers and the gpt model to compute the perplexity of a given sentence. import torch from transformers import GPT2LMHeadModel, GPT2TokenizerFast # You can change to gpt-large or other pretrained models that you can find in Huggingface. tokenizer = GPT2TokenizerFast.from_pretrained('distilgpt2') model = GPT2LMHeadModel....


Although, the previous answer is a good reference to find how to measure probability of a sentence using BERT, in order to perform a meaningful evaluation of cross-model (e.g., compare BERT with Roberta) they should use the same tokenization.


This may be best understood with a bit more of context from the article: A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the pretraining objective. Our current objective ...

Top 50 recent answers are included