Here I would show how we can use transformers and the gpt model to compute the perplexity of a given sentence.
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
# You can change to gpt-large or other pretrained models that you can find in Huggingface.
tokenizer = GPT2TokenizerFast.from_pretrained('distilgpt2')
model = GPT2LMHeadModel....
Although, the previous answer is a good reference to find how to measure probability of a sentence using BERT, in order to perform a meaningful evaluation of cross-model (e.g., compare BERT with Roberta) they should use the same tokenization.
This may be best understood with a bit more of context from the article:
A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether
autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
pretraining objective. Our current objective ...