# Validation loss not decreasing using dense layers altough training and validation data have the same distribution

I have a problem that I have great difficulties understanding the concept that leads to these results. I use a keras dense layer to map 13 input features to 3 output labels. During the training, the loss (RMSE) for the training data constantly decreases even after 2000 epochs. However, the loss for the validation data does not decrease any more after some 200 epochs as you can see in the figure.

Now the thing that I don't understand at all is how you can explain this outcome given that the training and validation data have (almost) exactly the same distribution for all of the 13 input features and the 3 output labels. They are drawn from the very same distribution. To illustrate that I have plotted the histograms for some input features and output labels (and the kernel density functions) as you can see here:

If you want to see the histograms for all input features and output labels you can find them here (in all the distribution of the trai, valid and test dataset look quite similar): https://filetransfer.io/data-package/nHxgDfvF#link

Also I calculated the correlation between each of the input features and the output labels for the training, validation and test dataset and the values are almost the same across the datasets for all combinations. If wou want to the see values you can find them here: https://filetransfer.io/data-package/iVruYbLx#link. This just further highlights, that the training and validation data have the same distribution.

Here you can find my code:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_percentage_error
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import BatchNormalization, Dense, Flatten
from matplotlib import pyplot as plt

ANN_input_data_features = ANN_input_data_features.values
ANN_input_data_labels = ANN_input_data_labels.values

# standardize input features X and output labels Y
scaler_standardized_X = StandardScaler()
ANN_input_data_features = scaler_standardized_X.fit_transform(ANN_input_data_features)

scaler_standardized_Y = StandardScaler()
ANN_input_data_labels = scaler_standardized_Y.fit_transform(ANN_input_data_labels)

#Split dataset into train, validation, an test
index_X_Train_End = int(0.7 * len(ANN_input_data_features))
index_X_Validation_End = int(0.9 * len(ANN_input_data_features))

X_train = ANN_input_data_features [0: index_X_Train_End]
X_valid = ANN_input_data_features [index_X_Train_End: index_X_Validation_End]
X_test = ANN_input_data_features [index_X_Validation_End:]

Y_train = ANN_input_data_labels [0: index_X_Train_End]
Y_valid = ANN_input_data_labels [index_X_Train_End: index_X_Validation_End]
Y_test = ANN_input_data_labels [index_X_Validation_End:]

#Train the model

numberOfInputFeatures = len(ANN_input_data_features[0])
numberOfOutputNeurons = len(ANN_input_data_labels[0])

model = keras.Sequential([
Flatten(input_shape=(numberOfInputFeatures,)),
Dense(30, activation='relu'),
#BatchNormalization(axis = 1),
Dense(50, activation='relu'),
#BatchNormalization(axis = 1),
Dense(50, activation='relu'),
#BatchNormalization(axis = 1),
Dense(30, activation='relu'),

keras.layers.Dense(numberOfOutputNeurons)])

entireFolderNameForTheResultsOfTheRun = "C:/Users/User1/Desktop/Training/"
pathOfTheFileForBestModel = entireFolderNameForTheResultsOfTheRun + "bestModelSingleTimeSlotTest.keras"
callbacks = [  keras.callbacks.ModelCheckpoint(pathOfTheFileForBestModel,  save_best_only=True) ]

history = model.fit(X_train, Y_train, epochs=2000, batch_size=10, validation_data=(X_valid, Y_valid), callbacks=callbacks)

# Predict the values from the test dataset
Y_pred = model.predict(X_test)

# Rescale the results of the predictions in the test dataset
Y_test_traInv = scaler_standardized_Y.inverse_transform(Y_test)
Y_pred_traInv = scaler_standardized_Y.inverse_transform(Y_pred)

#  Calculate the error in the test dataset
rms = mean_squared_error(Y_test_traInv, Y_pred_traInv, squared=True)
mape = mean_absolute_percentage_error(Y_test_traInv, Y_pred_traInv)

print("Evaluation with the test data")
print("Root Mean Squarred Error: ", rms)
print("Mean Absolute Percentage Error:", mape)

#Plot training results
plt.plot(history.history['mean_absolute_percentage_error'])
plt.plot(history.history['val_mean_absolute_percentage_error'])
plt.title('Mean absolute percentage errror')
plt.ylabel('Mean absolute percentage errror')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Loss function')
plt.ylabel('Mean squared Error')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()


Here is the data that I use (input features and output labels): https://filetransfer.io/data-package/ldU9KENV#link

I also tried it with batch normalization but this led to overall worse results. And I also played around with the number of layer, neurons, batchsize etc., but the problem remains the same or the results become way worse.

Now my question is how can this be explained. As far as I understand artifical neural networks, the networks adjusts its weight during the training such that it can explain the training dataset as well as possible. But if the training and validation dataset have the same distribution and the same correlation between input and outputs, as in my case, a model capable of explaining the traning dataset should also be able to explain the validation dataset which is not the case here.

Reminder: As I have still not received a satisfying answer, I would like to remind you on this question. I'll highly appreciate every further answer on this issue.

• This is a problem with the unbalanced dataset. You also do not indicate any activation function in the last layer, but this will come later. Your dataset is plenty of zeros and I am not sure your split respects such proportions. Apr 20 at 12:25
• @Oscar: Thanks a lot for your comment Oscar. Actually I now tried it with a linear activation function in the last layer and the results look exactly the same. About your other point (unbalaced dataset). As shown in the histograms, the distribution is almost identical for the train, validation (and test) dataset. Futher, I used the same data and made several runs randomly assigning rows to the train, validation and test dataset. And the results always look quite similar both regarding the histograms and - more importantly - regarding the training error and validation error. Apr 21 at 8:48
• @Oscar: Thanks for your comment. Any comments to my last comment? I'll highly appreciate every further comment from you. Apr 27 at 7:42
• @Oscar: Any further comment? May 2 at 8:46
• As said, your model is overfitting. Even if the distributions are identical if your model sees 10000 data corresponding to label 0 and 100 corresponding to other labels it will be less precise on the latter. Furthermore, linear activation function = no activation function, so it is useless, since your outputs are all positive, you can use relu. Finally, use regularisation techniques: dropout, minority label upsampling (what I would use in this case), etc May 2 at 12:58