LSTM networks can be used to generate new text. Given a sequence, I can predict the next word. Is there a way to get a score associated to each word predicted?
In particular, if the new word has never been seen by the LSTM network, can we train the LSTM to output a score of "no confidence"?
For example, this article gives the following example:
For example, let us consider the following three text segments:
1) Sir Ahmed Salman Rushdie is a British Indian novelist
and essayist. He is said to combine magical realism with
historical fiction.
2) Calvin Harris & HAIM combine their powers for a magical
music video.
3) Herbs have enormous magical power, as they hold the
earth’s energy within them.
Consider an LM that is trained on a dataset having the
example sentences given above — given the word “magical”,
what should be the most likely next word: realism, music, or power?
Say, that, in fact, the next word is neither one, but "power", but my LSTM has never seen that word before. So, the LSTM is going to predict one of the three words it has seen, but I would like it to output a low confidence score. Is this possible?