0
$\begingroup$

I have a problem that I have great difficulties understanding the concept that leads to these results. I use a keras dense layer to map 13 input features to 3 output labels. During the training, the loss (RMSE) for the training data constantly decreases even after 2000 epochs. However, the loss for the validation data does not decrease any more after some 200 epochs as you can see in the figure.

enter image description here

Now the thing that I don't understand at all is how you can explain this outcome given that the training and validation data have (almost) exactly the same distribution for all of the 13 input features and the 3 output labels. They are drawn from the very same distribution. To illustrate that I have plotted the histograms for some input features and output labels (and the kernel density functions) as you can see here:

enter image description here enter image description here

If you want to see the histograms for all input features and output labels you can find them here (in all the distribution of the trai, valid and test dataset look quite similar): https://filetransfer.io/data-package/nHxgDfvF#link

Also I calculated the correlation between each of the input features and the output labels for the training, validation and test dataset and the values are almost the same across the datasets for all combinations. If wou want to the see values you can find them here: https://filetransfer.io/data-package/iVruYbLx#link. This just further highlights, that the training and validation data have the same distribution.

Here you can find my code:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_percentage_error
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import BatchNormalization, Dense, Flatten
from matplotlib import pyplot as plt

#Read data from  csv files
ANN_input_data_features = pd.read_csv("C:/Users/User1/Desktop/TestDataANN_InputFeatures.csv", sep=';')
ANN_input_data_labels = pd.read_csv("C:/Users/User1/Desktop/TestDataANN_OutputLabels.csv", sep=';')
ANN_input_data_features = ANN_input_data_features.values
ANN_input_data_labels = ANN_input_data_labels.values

            
# standardize input features X and output labels Y
scaler_standardized_X = StandardScaler()
ANN_input_data_features = scaler_standardized_X.fit_transform(ANN_input_data_features)

scaler_standardized_Y = StandardScaler()
ANN_input_data_labels = scaler_standardized_Y.fit_transform(ANN_input_data_labels)


#Split dataset into train, validation, an test
index_X_Train_End = int(0.7 * len(ANN_input_data_features))
index_X_Validation_End = int(0.9 * len(ANN_input_data_features))

X_train = ANN_input_data_features [0: index_X_Train_End]
X_valid = ANN_input_data_features [index_X_Train_End: index_X_Validation_End]
X_test = ANN_input_data_features [index_X_Validation_End:]

Y_train = ANN_input_data_labels [0: index_X_Train_End]
Y_valid = ANN_input_data_labels [index_X_Train_End: index_X_Validation_End]
Y_test = ANN_input_data_labels [index_X_Validation_End:]


#Train the model
optimizer_adam = tf.keras.optimizers.Adam(learning_rate= 0.001)

numberOfInputFeatures = len(ANN_input_data_features[0])
numberOfOutputNeurons = len(ANN_input_data_labels[0])

model = keras.Sequential([
    Flatten(input_shape=(numberOfInputFeatures,)),
    Dense(30, activation='relu'),
    #BatchNormalization(axis = 1),
    Dense(50, activation='relu'),
    #BatchNormalization(axis = 1),
    Dense(50, activation='relu'),
    #BatchNormalization(axis = 1),
    Dense(30, activation='relu'),

    keras.layers.Dense(numberOfOutputNeurons)])

entireFolderNameForTheResultsOfTheRun = "C:/Users/User1/Desktop/Training/"
pathOfTheFileForBestModel = entireFolderNameForTheResultsOfTheRun + "bestModelSingleTimeSlotTest.keras"
callbacks = [  keras.callbacks.ModelCheckpoint(pathOfTheFileForBestModel,  save_best_only=True) ]

model.compile(loss="mean_squared_error", optimizer=optimizer_adam, metrics=['mean_absolute_percentage_error'])
history = model.fit(X_train, Y_train, epochs=2000, batch_size=10, validation_data=(X_valid, Y_valid), callbacks=callbacks)

# Predict the values from the test dataset
model = keras.models.load_model(pathOfTheFileForBestModel)
Y_pred = model.predict(X_test)

# Rescale the results of the predictions in the test dataset
Y_test_traInv = scaler_standardized_Y.inverse_transform(Y_test)
Y_pred_traInv = scaler_standardized_Y.inverse_transform(Y_pred)


#  Calculate the error in the test dataset
rms = mean_squared_error(Y_test_traInv, Y_pred_traInv, squared=True)
mape = mean_absolute_percentage_error(Y_test_traInv, Y_pred_traInv)

print("Evaluation with the test data")
print("Root Mean Squarred Error: ", rms)
print("Mean Absolute Percentage Error:", mape)


#Plot training results
plt.plot(history.history['mean_absolute_percentage_error'])
plt.plot(history.history['val_mean_absolute_percentage_error'])
plt.title('Mean absolute percentage errror')
plt.ylabel('Mean absolute percentage errror')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Loss function')
plt.ylabel('Mean squared Error')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

Here is the data that I use (input features and output labels): https://filetransfer.io/data-package/ldU9KENV#link

I also tried it with batch normalization but this led to overall worse results. And I also played around with the number of layer, neurons, batchsize etc., but the problem remains the same or the results become way worse.

Now my question is how can this be explained. As far as I understand artifical neural networks, the networks adjusts its weight during the training such that it can explain the training dataset as well as possible. But if the training and validation dataset have the same distribution and the same correlation between input and outputs, as in my case, a model capable of explaining the traning dataset should also be able to explain the validation dataset which is not the case here.

Reminder: As I have still not received a satisfying answer, I would like to remind you on this question. I'll highly appreciate every further answer on this issue.

$\endgroup$
11
  • $\begingroup$ This is a problem with the unbalanced dataset. You also do not indicate any activation function in the last layer, but this will come later. Your dataset is plenty of zeros and I am not sure your split respects such proportions. $\endgroup$
    – Oscar
    Apr 20, 2022 at 12:25
  • $\begingroup$ @Oscar: Thanks a lot for your comment Oscar. Actually I now tried it with a linear activation function in the last layer and the results look exactly the same. About your other point (unbalaced dataset). As shown in the histograms, the distribution is almost identical for the train, validation (and test) dataset. Futher, I used the same data and made several runs randomly assigning rows to the train, validation and test dataset. And the results always look quite similar both regarding the histograms and - more importantly - regarding the training error and validation error. $\endgroup$
    – PeterBe
    Apr 21, 2022 at 8:48
  • $\begingroup$ @Oscar: Thanks for your comment. Any comments to my last comment? I'll highly appreciate every further comment from you. $\endgroup$
    – PeterBe
    Apr 27, 2022 at 7:42
  • $\begingroup$ @Oscar: Any further comment? $\endgroup$
    – PeterBe
    May 2, 2022 at 8:46
  • $\begingroup$ As said, your model is overfitting. Even if the distributions are identical if your model sees 10000 data corresponding to label 0 and 100 corresponding to other labels it will be less precise on the latter. Furthermore, linear activation function = no activation function, so it is useless, since your outputs are all positive, you can use relu. Finally, use regularisation techniques: dropout, minority label upsampling (what I would use in this case), etc $\endgroup$
    – Oscar
    May 2, 2022 at 12:58

1 Answer 1

0
$\begingroup$

Your model is overfitting the train data.
Please use Dropout or any other technique to counter overfitting.

This is what I got with Dropout in 250 epochs

enter image description here

Code link- Colab scratch NB

$\endgroup$
6
  • $\begingroup$ Thanks a lot for your comment. The actual question is why can an overfitting happen in my example? As stated in my question, the training data and the validaiton data have exactly the same distribution and properties. All input features and output labels have almost the same distribution and the correlation between the inputs and outputs are identical for the training and test dataset. So a model capable of explaining the training data should also be able to explain and predict the validation datas. When the training error decreases, the validaiton error should decrease likewise. $\endgroup$
    – PeterBe
    Apr 25, 2022 at 13:04
  • $\begingroup$ Thanks for your comment. Any comments to my last comment? I'll highly appreciate every further comment from you. $\endgroup$
    – PeterBe
    Apr 27, 2022 at 7:42
  • $\begingroup$ Thanks for your answer and effort 10xAI. I really appreciate it. Well your suggested solution with the droupout leads to a higher error for the validation dataset compared to my approach. I am having a RMSE of about 0,4 while your suggested approach yields to a RMSE of around 0,6. So I think it makes it worse. What do you think about this? $\endgroup$
    – PeterBe
    May 5, 2022 at 8:00
  • 1
    $\begingroup$ With overfit, you can even get 0 MSE. You should accept the generalized value. Maybe 0.6 in this case. $\endgroup$
    – 10xAI
    May 6, 2022 at 4:25
  • $\begingroup$ Thanks for your comment. Actually I am talking about the validaiton dataset. If the RMSE is quite low there it means that the model is in fact able to generalize. In my approach the RMSE is way lower than in yours for the validation dataset which means that my model can generalize better as it yields to a lower error on unseen data. $\endgroup$
    – PeterBe
    May 6, 2022 at 7:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.