I am looking for a model with which I can predict the probability of a current word given its n predecessors (or successors) in a sentence.

Please note: I do not want to generate text nor do I want to predict the next word, but rather I want to know if a given already existing word/sentence makes sense or not.

For this I am looking for a solution that solves the following two problems:

First: For example: Given the sentences "I build a house" and "I build a soup".

I want to have a model that tells me P( "house" | "I build a") and P( "soup" | "I build a")

So in this illustrative example I would like to get something like 30% for "house" and say 0.1% for "soup".

Second: This is related to the first requirement but now I want to ask what is the probability of any word in a sentence given the other words.

For example given again the sentence "I build a house" what is P("build" | "I", "a house"). In this case I would expect the model to tell me that this word is reasonable within this context. While P("build" | "I", "a soup") should be evaluated as unreasonable.

Which model would be recommendable for solving this problem? Ideally a pretrained one that is available for download.


1 Answer 1


What you are describing is respectively:

  • a normal (causal) language model (e.g. GPT-2), which computes the probability of a token based on the previous tokens.
  • a masked language model (e.g. BERT), which computes the probability of a masked token in the middle of the sentence.

Beware, however, that modern language models don't tend to use words as tokens. Instead, they use subword tokens, where a single word can be composed of multiple tokens. Therefore, you would probably need to train your own word-level causal and masked language models.

  • $\begingroup$ Thanks for the answer and the note on subwords. Regarding the answer, I have a follow up answer: Do you know if I can use a model from Huggingface (eg. pretrained BERT model) to do this? $\endgroup$
    – toom
    Aug 2 at 12:31
  • $\begingroup$ I am not aware of any word-level models in Huggingface; you would need to look for one. Alternatively, you may use GPT-2 (causal) and BERT (masked) and combine the probabilities of the subwords of the word. Formally, the probabilities you obtain would not be correct but, in practical terms, it may be enough. $\endgroup$
    – noe
    Aug 2 at 13:17
  • $\begingroup$ I understand the idea. But as I understand it BERT will just predict/suggest the most appropriate word. However, I do not want a word, but rather I want the probability of a given word given a certain context which is very different from the original usecase of BERT. $\endgroup$
    – toom
    Aug 2 at 13:23
  • $\begingroup$ BERT gives a probability distribution over the token space. It literally gives you the probability of the gap being filled with each possible token. $\endgroup$
    – noe
    Aug 2 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.