How can I get probabilities of next word with ELMO?

Question

ELMO is a language model, build to to compute the probability of a word, given some prior history of words seen. How can I get this probability from pretained ELMO model?

Answer 1

ELMO cannot be used to predict the next word.

The reason is that ELMO is like two language models combined: one normal language model that predicts the next word based on the previous ones and another language model in the reverse direction. The hidden representations obtained from both directions are combined to generate the word predictions.

This way, ELMo cannot be used as is for next word prediction because it is designed to receive the whole sentence and is not trained in a causal language model loss.

In order to select a model for next word prediction, you should focus in normal (causal) language models, not bidirectional LMs (e.g. ELMo), masked LMs (e.g. BERT, XLM) or permutation LMs (e.g. XLNet).

Some examples of popular causal language models are OpenAI's GPT and GPT-2, or Transformer-XL. I would recommend taking a look at Hugging Face's PyTorch transformers github repo, which makes it super easy to load pretrained weights for renowned LMs

Answer 2

ELMO is not a language model. As specified in the paper:

We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy).

ELMO representations (embeddings) are indeed calculated by looking at the context of the word. But the output is a fixed size vector that contains/represents the contextualised meaning of that word. This can be compared to other word embeddings such as word2vec, glove or fasttext but with the novelty of having different representations of the same word based on their context within a sentence.

I suggest you do some reading about the differences about language models and word representations.