4
$\begingroup$

According to article about LSTM here, I know that:

it allows us to score arbitrary sentences based on how likely they are to occur in the real world. This gives us a measure of grammatical and semantic correctness. Such models are typically used as part of Machine Translation systems.

But, it seems that this article doesn't point out how to compute the score with LSTM.

Any way to compute the score?

$\endgroup$
6
$\begingroup$

Typically you would use a perplexity value. For example, if your LSTM model is word-based and you have a sentence $[x_1, x_2 . . . x_N]$, and your model predicts the words that appear in that sentence with probability $p(x_i|x_0..x_{i-1})$ (where $x_0$ is a "start token" or whatever you use to start your RNN prediction sequence). Then you might quote a per-word perplexity for that sentence under your model as

$- \frac{1}{N}\sum_{i=1}^N log_2(p(x_i|x_0..x_{i-1}))$

Using an LSTM to predict consecutive words, it is practical to construct an array of probabilities $[p_1, p_2 . . . p_N]$ by running the network on the sentence and noting the probabilities for the correct matching word - i.e. $p_i = $ the predicted probability of the correct class $x_i$ at each step, which simplifies the expression:

$- \frac{1}{N}\sum_{i=1}^N log_2(p_i)$

| improve this answer | |
$\endgroup$
  • $\begingroup$ sorry, I have a stupid question, how cound I get $[p_1, p_2 . . . p_N]$ ? I find a LSTM implementation, its next_word_probs is a matrix(see here). Or you mean perplexity like this $\endgroup$ – roger Jun 14 '17 at 12:57
  • $\begingroup$ The matrix is the probability for every word on each time step, you need the probability of the word from the sentence you are scoring (you should have a way to convert words to indexes, and you know the time step, so this is just an index lookup). Yes the test_perplexity is likely to be a similar measure to that given in my answer - there are a few different ways to do it, so might not be 100% same value, but if you can evaluate your target sentence using the same test code, then that should be equally good for you to score your sentence, and would save you effort to implement it. $\endgroup$ – Neil Slater Jun 14 '17 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.