1. Should we convert our inputs to on-hot-vectors and expect one-hot-vectors as output? I mean can we feed LSTM with a vector like x=[12, -234, 54 , 78 , 12 , 6], and have a label vector like this: y=[13, -230, 50, 80 , 9 , 7]? (And we don't use one-hot-vectors at all). Will such network work properly? Or it's better to convert inputs/outputs to a one-hot-vector and this is essence of LSTM?

  2. If feeding LSTM with one-hot-vector isn't a necessary rule, and we like to feed our network with such vectors in my previous question, should we again use softmax() function for out outputs? Or we can use better options for such problem(or even don't use any functions there)? If we must(or better) to use softmax, how can we interpret it's result?

  3. If it's better to convert our inputs/outputs to one-hot-vectors, can we use two or three hot vectors(I mean: x =[1,0,0,1,0,0] or x=[0,1,1,1,0,0])? Does this work properly or it disrupts the LSTM performance?


2 Answers 2

  1. This depends on what your data is representing and what you want to predict. My understanding of One-Hot-Encoding is that this should only be used for encoding of categorical features. For example, if you have a feature representing a category of K classes, you should one hot encode this as well as the Y variable (if you are trying to predict this categorical variable). Of course, have the final layer be a softmax to output a distribution of size K.

  2. This highly depends on what your data is representing. If categorical, see above. If just a simple numeric, you should not one hot encode this. You could, I guess, if this set of integers is finite and small, but there is no need to learn the extra weights. You should only be using a softmax when you want to output a vector of K dimension where the entries all sum to one (perfect for representing a probability distribution over K classes). The final layer should output whatever it is you want to predict. If you want to predict something as simple as a numeric variable at the next time-step, just have a dense layer of size 1, with some activation function (relu probably). More information about what exactly you are trying to predict is needed to know how to recommend anything concrete here.

  3. I'm not sure about this. You could have inputs and outputs represented this way, but you wouldn't use a softmax activation at the end. You would use a dense layer outputting a vector the size of your X variable, that is, if that's the variable you are trying to predict. I'm not super well trained in various types of use cases for LSTMs, but from my experience, I can't think of a reason to do this.

  • $\begingroup$ Thank you very much kylec, unfortunately I have not enough reputations to give you a +vote at this time. BTW, about your question in part 2, I am trying to do stock price prediction, and by [200 210 199 205 100 etc.] I mean [open high low close volume etc.] numbers. What do you think about it? I am confused that what input/output vectors should I have for this porpuse? Also what activation functions? $\endgroup$ Commented Feb 21, 2019 at 8:28
  • $\begingroup$ All good! I'm still a newbie to StackOverflow (well, actually participating, at least) so I'm restricted as well. I'm assuming all of your variables are numeric, so if you are trying to predict all these numerical features (essentially have a prediction for the entire next time-step), have your Y variable be the next time-step values for each variable, and at the end of your network have a Dense layer with units equal to however many numeric variables you are trying to predict. I'd start with a relu activation for this layer. $\endgroup$
    – kylec123
    Commented Feb 21, 2019 at 20:40
  1. In my experience, one-hot encoding helps the model to distinguish differences in data where small changes would have large outcomes in the result. Say, if it's pixel data of an image, a small change in one pixel would have little or no effect on what the image represents, so the model can easily differentiate that data without one-hot encoding it.

  2. The final layer's activation function should be chosen based on the kind of data you're trying to model. Use no activation (a.k.a. linear) and MSE loss if you just want it to predict numbers with no restriction. For example, if the model needs to be able to predict the number 1000, then let it use linear activation in the final layer. Refrain from using ReLU here if you need the model to also be able to output negative numbers. Use softmax activation with categorical cross-entropy loss if the model is intended to make a prediction on what the best "choice" is for something. This almost always takes the form of classification. Use sigmoid activation with binary cross-entropy loss if the model is to answer a bunch of "yes or no" questions about the data. For example, a model takes text as input and outputs multiple numbers between 0 and 1, and perhaps the first number indicates whether or not the text has a positive connotation, and maybe the second number indicates whether or not the text uses a wide range of vocabulary. This is what we call binary classification, and we're completely allowed to have multiple "classes" for it. The main difference between sigmoid and softmax is that softmax normalizes the sum of the output whereas sigmoid does not. So softmax makes the output behave like a probability distribution, and sigmoid makes each number in the output be independent of one another, but still behaving as though it's a probability of "yes" for that specific part of the label.

  3. Having said that, use binary classification (sigmoid activation and binary cross-entropy loss) if the labels in your data set are vectors consisting of only zeros and ones but are not restricted to being one-hot vectors.

For a final note, these rules apply to almost any neural network architecture, including LSTM/GRU and convolutional neural networks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.