0
$\begingroup$

LSTM networks can be used to generate new text. Given a sequence, I can predict the next word. Is there a way to get a score associated to each word predicted?

In particular, if the new word has never been seen by the LSTM network, can we train the LSTM to output a score of "no confidence"?

For example, this article gives the following example:

For example, let us consider the following three text segments:

1) Sir Ahmed Salman Rushdie is a British Indian novelist
and essayist. He is said to combine magical realism with
historical fiction.

2) Calvin Harris & HAIM combine their powers for a magical
music video.

3) Herbs have enormous magical power, as they hold the
earth’s energy within them.

Consider an LM that is trained on a dataset having the
example sentences given above — given the word “magical”,
what should be the most likely next word: realism, music, or power?

Say, that, in fact, the next word is neither one, but "power", but my LSTM has never seen that word before. So, the LSTM is going to predict one of the three words it has seen, but I would like it to output a low confidence score. Is this possible?

$\endgroup$
4
  • 1
    $\begingroup$ Couldn't you require your softmax output to exceed a threshold for the prediction or you call it "not confident"? $\endgroup$
    – Wayne
    Mar 22, 2018 at 20:58
  • $\begingroup$ @Wayne Say you are using Keras, so you have model.add(Dense(classes, activation='softmax')), how would you implement what you suggest? $\endgroup$
    – user
    Mar 26, 2018 at 1:31
  • $\begingroup$ I have not tried word prediction, but the softmax gives you a value for each output, where all of those values sum to 1. So pick the column with the highest value as your predicted word and use that value to determine your confidence. (It may be fairly low, since there might be a small chance of lots of words and then larger chances of your most-common three.) $\endgroup$
    – Wayne
    Mar 27, 2018 at 13:00
  • $\begingroup$ @Wayne, you are right! It was simple. If you make it an answer I will accept it. $\endgroup$
    – user
    Mar 28, 2018 at 1:06

1 Answer 1

1
$\begingroup$

Couldn't you require your softmax output to exceed a threshold for the prediction or you call it "not confident"?

I have not tried word prediction, but the softmax gives you a value for each output, where all of those values sum to 1. So pick the column with the highest value as your predicted word and use that value to determine your confidence. (It may be fairly low, since there might be a small chance of lots of words and then larger chances of your most-common three.)

$\endgroup$
1
  • $\begingroup$ You can also take the value of Shannon's entropy of softmax output, this can be a better measure of confidence. As whenever distribution is peaky entropy will be low, and if distribution is nearly uniform then entropy will be high. Lower the entropy, higher the confidence. $\endgroup$
    – mdilip
    Nov 6, 2018 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.