Why is predicted rainfall by LSTM coming negative for some data points?

Question

I have used supervised learning with LSTM network using tanh activation function and 0.1 dropout for time series prediction.my loss='mean_squared_error', optimizer='adam'. The predicted time series is shown below where x axis shows future months and y axis shows rainfall in mm.Orange line is predicted and blue line is actual. Some points are being predicted below 0 (negative) even though training dataset has all points having values >=0. My training dataset is of 64 years (all positive data) and I am predicting for 12 years (144 data points shown)

Model code using ReLU activation:

model = Sequential()
# LSTM model
model.add(LSTM(128, batch_input_shape=(batch_size, look_back, 1),activation='relu', stateful=True, return_sequences=False))
model.add(Dropout(0.1)) #for better regularization
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')

Answer 1

Currently your model is not restricted from guessing any number. In this case, you may want to use something like a ReLU activation function to restrict your output domain.

The ReLU activation needs to be added to the last layer of your model. If you add it to an intermediate layer but not the last layer then your model can still output negative numbers, e.g. in the case of the last layer having a negative weight.

Note this is not really a new idea. There are strong similarities between logistic regression and what we’re doing here. In both cases we want to limit the possible guesses, in your case to non-negative numbers and in logistic regression‘s case to [0, 1].