2
$\begingroup$

I have used supervised learning with LSTM network using tanh activation function and 0.1 dropout for time series prediction.my loss='mean_squared_error', optimizer='adam'. The predicted time series is shown below where x axis shows future months and y axis shows rainfall in mm.Orange line is predicted and blue line is actual. Some points are being predicted below 0 (negative) even though training dataset has all points having values >=0. My training dataset is of 64 years (all positive data) and I am predicting for 12 years (144 data points shown) Rainfall prediction

Model code using ReLU activation:

model = Sequential()
# LSTM model
model.add(LSTM(128, batch_input_shape=(batch_size, look_back, 1),activation='relu', stateful=True, return_sequences=False))
model.add(Dropout(0.1)) #for better regularization
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
$\endgroup$
  • 3
    $\begingroup$ Because your model is not aware of the non-negative constraint. You could try using a non-negative transformation like $\exp$ or $\max(x,0)$. Welcome to the site. $\endgroup$ – Emre Aug 13 '18 at 19:57
  • $\begingroup$ Emre, that is an answer not a comment. $\endgroup$ – kbrose Aug 13 '18 at 22:50
  • $\begingroup$ So if I use relu activation function, will this problem get solved? $\endgroup$ – Roy Aug 14 '18 at 7:11
3
$\begingroup$

Currently your model is not restricted from guessing any number. In this case, you may want to use something like a ReLU activation function to restrict your output domain.

The ReLU activation needs to be added to the last layer of your model. If you add it to an intermediate layer but not the last layer then your model can still output negative numbers, e.g. in the case of the last layer having a negative weight.

Note this is not really a new idea. There are strong similarities between logistic regression and what we’re doing here. In both cases we want to limit the possible guesses, in your case to non-negative numbers and in logistic regression‘s case to [0, 1].

$\endgroup$
  • $\begingroup$ Thanks. ReLU should work then, I will try and let you know. $\endgroup$ – Roy Aug 14 '18 at 18:44
  • $\begingroup$ I tried using ReLU but in that case while training, after some epochs, the loss is all of a sudden going to NA. I am not sure why this is happening? $\endgroup$ – Roy Aug 28 '18 at 6:32
  • $\begingroup$ Are you doing division anywhere in your model? $\endgroup$ – kbrose Aug 28 '18 at 14:02
  • $\begingroup$ If you share your architecture it will be easier to help. $\endgroup$ – kbrose Aug 28 '18 at 14:03
  • $\begingroup$ model = Sequential() #LSTM model model.add(LSTM(128, batch_input_shape=(batch_size, look_back, 1),activation='relu', stateful=True, return_sequences=False)) model.add(Dropout(0.1)) #for better regularization model.add(Dense(1)) model.compile(loss='mean_squared_error', optimizer='adam') $\endgroup$ – Roy Aug 29 '18 at 3:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.