0
$\begingroup$

I'm wondering how continuous output values of deep learning networks are converted to byte values or other discrete values for that sake.

For example here: In his famous article The Unreasonable Effectiveness of Recurrent Neural Networks, Andrej samples Shakespeare texts using an LSTM. An LSTM's output consists of sigmoid outputs multiplied by tanh outputs:

enter image description here (Understanding LSTM Networks, Christopher Olah, 2015)

These values are always in the interval [-1, 1], so their product (and hence the output of the LSTM) is also in this interval. How are these values converted to byte values {0, 1, 2, 3, ..., 255}?

enter image description here

$\endgroup$
1

1 Answer 1

2
$\begingroup$

The typical approach to have neural networks generate discrete outputs is to make them generate a probability distribution over the space of options. Then, you can choose the option with the highest probability, or use a different sampling strategy (i.e. beam search in text generation).

To generate a probability distribution over N options, you project the output of your model (i.e. an LSTM) to an N-dimensional space and then apply the softmax function to normalize the N components and make them add up to 1. The N-dimensional vector before the softmax is referred to as unnormalized log probabilities or "logits".

The usual loss function for this kind of discrete output is the categorical cross-entropy.

This kind of approach is used not only for text generation, but also in multi-class classification.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.