Output a word instead of a vector after word embedding?

Question

I'm trying to play around with a toy implementation of translation or text-summarization. I understand now that most people use an embedding layer before whatever model they use, which produces something like 300-dimensional vector. But what does the model output? Like for a encoder-decoder model, it's inputs are sequences of those vectors. Then what does the final layer of decoder come out with? Not like event extraction or something like that, which we classify to a small number of classes.

So my core question is: Is the output also a 300-dimensional vector, which then I have to generate the word based on the most similar word vector? Or is it the index of a word from the original word space? In the second case, the network is classifying to more than 100000 classes?

And also, is there any existing package that support this "reverse embedding to word"?

I have not seen any question alike on the site. Please mark it as duplicate if it is.

Any help would be appreciated.

Answer 1

This answer describes how you go from a vector in the embedding space back to the the most similar class (e.g. word or character).

Answer 2

It depends on the architecture of the model.

A lot of work uses the text data to classify some specific category (e.g., the IMDB movie reviews or sentiment analysis on Tweets).

Other work can actually take the words of a sentence and predict the last word. Or break it into each word predicting the subsequent word, which is really what the RNN/embedding dimension is doing.

Other's have made these even at the character level. Ultimately, it depends on how you process the data and specify your outcome. Below are some useful tutorials that should add a little more clarity.

1. http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/

2. http://karpathy.github.io/2015/05/21/rnn-effectiveness/

3. http://deeplearning.net/tutorial/lstm.html

4. http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/

5. https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py

Answer 3

The actual answer: yes you have a massive output layer with 100000 classes and take a softmax. You don’t look up the word via embedding in general.