why we sample when predicting with Recurent Neural Network

Question

I trained a Recurrent Neural Network to predict the next word in a sentence.

I trained and now I want to predict, but there is something I am not getting well.

I saw it in many tutorials even in the Keras documentation.

When we want to predict, we get the probabilities from the predict function and we pass it to the sample function bellow.

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)

I am not getting what this function is doing.

I know that :

My predictions are an array of probabilities and from that, the word indice is the word with the largest probability.

From the function I can see that they are doing log, and then exp, that is what I am not getting.

Why are they doing that?

I saw this in many tutorials but no one explained it well.

Probably I should go back to my statistics and probabilities courses.

Please point me to the right direction.

Answer 1

It seems to me you have two questions:

• Why use sampling to generate text from a trained RNN language model?
• How does this particular sampling function from Keras work?

Why use sampling to generate text from a trained RNN language model

After training a language model, very often you would like to use the model to generate new text. For a word-level RNN language model, text is generated one word at a time. In each step, the model outputs a probability distribution over the entire vocabulary. With a toy vocabulary such as:

$$ vocabulary=\hspace{0.2cm} \begin{bmatrix} cat \\ dog \\ frog \end{bmatrix} $$

the distribution output by the model at each generation step could look like

$$ predictions=\begin{bmatrix} 0.7 \\ 0.1 \\ 0.2 \end{bmatrix} $$

each entry in the vector corresponds to one entry in our vocabulary.

After that, it is up to the user how to select a word from this distribution as the next word in the generated text. Two very different methods to pick a word from the distribution are:

1. Greedy search: take the word with the highest probability (will pick "cat")
2. Sampling: sample from the distribution, take into account probabilities (will pick a random word, but "cat" has the highest chance of being picked)

So, searching will return a very probable string (and the same one every time we try), while sampling will return somewhat likely, but more varied text. Often, this means that sampling leads to more "interesting" text.

How does this particular sampling function work?

As far as I can tell, the input for this sampling function is a vector of normalized probabilities after softmax, such as

np.array([0.7, 0.1, 0.2])

Here are some explanations about each line:

preds = np.asarray(preds).astype('float64')

This just makes sure that other array-like objects, such as Python lists, work as input, but are converted to the correct numpy array type.

preds = np.log(preds) / temperature

preds is a probability distribution that was created by applying the softmax function to the final output of the model network (those final values are called logits). Logits are assumed to be unnormalized log probabilities, applying softmax normalizes them.

This is a problem for softmax with temperature, because it is the unnormalized logits that are supposed to be divided by the temperature:

$$ q_i = \frac{exp(z_i/T)}{\sum_j{exp(z_j/T)}} $$

So, np.log(preds) just moves the values back into logspace. Without this, we would not be scaling the actual network outputs, but exponentiated values.

exp_preds = np.exp(preds)
preds = exp_preds / np.sum(exp_preds)

Those two lines just apply the softmax function, regardless of what happens before or after.

probas = np.random.multinomial(1, preds, 1)

This function with those particular arguments is sampling a one-hot vector such as

$$ predictions=\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} $$

where the probability of each index being the 1 is dictated by preds.

return np.argmax(probas)

Finally, this returns the index of the largest element in the vector. This index will correspond to one word in our vocabulary.

---

I hope this helps you understand the matter, or perhaps express more clearly what you would need additional explanations for.

Answer 2

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    # convert input to float64 array
    preds = np.asarray(preds).astype('float64') 
    #devide the logarythm of the predictions by temperature
    preds = np.log(preds) / temperature 
    # calculates e^preds for each pred in your list where e is Euler's number (approximately 2.718)
    exp_preds = np.exp(preds)
    # divide euler ^ xn / sum of euler ^xn
    preds = exp_preds / np.sum(exp_preds)
    # run random experiment eg: -> array([[4, 1, 7, 5, 2, 1]])
    probas = np.random.multinomial(1, preds, 1)
    # pull the highest value of array eg 7 (with the example above)
    return np.argmax(probas)