LSTM evaluation metric MAE explanation

I have a hard time understanding the LSTM model performance as I summarize my model as follow:

X_train.shape
(120, 7, 11)
y_train.shape
(120,)
X_test.shape
(16, 7, 11)
y_test.shape
(16,)

model = keras.Sequential()
model.add(keras.layers.LSTM(100, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences = True))
model.add(keras.layers.Dropout(rate = 0.2))
model.add(keras.layers.LSTM(20))
model.add(keras.layers.Dropout(rate = 0.2))
model.add(keras.layers.Dense(1))
model.compile(loss='mean_squared_error', optimizer=keras.optimizers.Adam(0.001), metrics = ['mae'])

history = model.fit(
X_train, y_train,
epochs=60,
batch_size=5,
verbose= 0,
validation_split = 0.1,
shuffle=False
)


Based on the below plots, both MSE and MAE decrease in the training process and their corresponding values are near zero.

However the prediction is not precise enough as I realize:

y_pred = model.predict(X_test)
model.evaluate(X_test,y_test)
[0.04673878103494644, 0.15574690699577332]


So my question is, how does really my model perform? I mean how can interpret its performance,since both MSE and MAE seem to be low but the prediction values are not quite convincing.

1 Answer

You are getting loss near to 0 but, Your true distribution of y in the range of 0-1 so, that 0.04 loss may be high loss. Just get random model and check the loss. You will get to know how much you decreased the loss. I will suggest to use r^2metric for evaluation.